Seminar | Bot or Not? The new age of false profits

Generative AI (GenAI) is now a staple in the cybercriminal’s toolkit. In 2024 the UK-based multinational, Arup lost £20m to a sophisticated GenAI scam while in the US, election security faced threats from AI spoofs of Joe Biden. However, the media focus on extreme cases overlooks its use in countless smaller frauds where malicious GenAI harms the ordinary public directly and at scale.

GenAI’s success in cybercrime hinges on its output quality – particularly the credibility of the voice and linguistic content – to convince victims the interaction is legitimate. This talk presents two versions of the “Bot or Not” online perception tests designed as games. To date, these collectively have over 1,000 submissions that reveal patterns of vulnerabilities in the public’s ability to detect GenAI. Specifically, Bot or Not: Text Edition tests the detection of AI-generated text, while Bot or Not: Audio Edition tests our ability to recognise AI-generated voice. This work compares participant performance across modes to reveal the strengths and weaknesses of both synthetic language and human perception, and concludes with some thoughts about the diminishing window of opportunity left for the detection of malicious genAI content.