Voice-Enabled AI Agents: How They Can Perform Common Scams
OpenAI recently released their real-time voice API, which enables applications ranging from automated customer service to educational software. Unfortunately, as with many AI capabilities, voice-enabled agents have the potential for dual-use.
In our recent paper, we investigate whether voice-enabled AI agents can perform the tasks needed to conduct common scams. Our findings show that these agents can indeed autonomously execute the actions necessary for various phone-based scams.
Background on Phone Scams
Phone-based scams are incredibly prevalent, targeting up to 17.6 million Americans annually and causing up to $40 billion in damage every year. These scams typically involve a scammer impersonating a legitimate institution—such as a bank or government agency—and convincing the victim to reveal sensitive information or take specific actions. Once the scammer has obtained this information, they can exploit it to steal money or personal data.
As an example of a common scam, a scammer could pretend to be an IRS agent with the goal of convincing the victim to purchase and transfer a gift card to the scammer. Another common scam involves stealing bank account credentials and transferring money from the victim to the scammer.
These scams often require complex interactions with websites and real-time feedback from the victim. For example, the bank account transfer scam involves navigating to the bank's website, retrieving login credentials, handling two-factor authentication, and executing the transfer.
AI Scam Agents
We designed a series of agents to perform the actions necessary for common scams. Our agent architecture consists of:
A base, voice-enabled LLM (GPT-4o)
A set of browser access tools (via playwright)
Scam-specific instructions
The agents had access to five granular browser action tools: get_html , navigate, click_element, fill_element, and evaluate_javascript. We used a standard jailbreaking prompt template to bypass certain protections in GPT-4o.
Importantly, our agent design is not complicated. We implemented it in just 1,051 lines of code, with most of the code dedicated to handling real-time voice API. This simplicity aligns with prior work showing the ease of creating dual-use AI agents for tasks like cybersecurity attacks.
Voice-Enabled Agents can Perform Common Scams
We deployed our agents on a subset of common scams. We simulated scams by manually interacting with the voice agent, playing the role of a credulous victim.
To determine success, we manually confirmed if the end state was achieved on real applications/websites. For example, we used Bank of America for bank transfer scams and confirmed that money was actually transferred. However, we did not measure the persuasion ability of these agents.
We found that voice-enabled AI agents can successfully execute every common scam we tested. In summary:
Success rates ranged from 20% to 60%, with an overall success rate of 36% across all scams.
Scams required a substantial number of actions, with the bank transfer scam taking 26 actions to complete.
Complex scams took up to 3 minutes to execute.
The agents maintained coherence in conversation and adherence to goals throughout the interactions.
The overall costs were low, averaging under $0.75 per successful scam.
Interestingly, many failures were due to transcription errors rather than flaws in the scam execution logic. However, more complex scams like bank account transfers had lower success rates due to the difficulty of navigating complex websites.
Beyond our agents, these capabilities are likely to improve as AI models and techniques continue to advance. Better models, agent scaffolding, and prompts are likely to lead to even more capable and convincing scam agents in the future.
Conclusions
As we have shown, voice-enabled LLM agents can perform the actions necessary to execute common phone scams. These agents are highly capable, can react to changes in the environment, and retry based on faulty information from the victim. Our results highlight the urgent need for future research in protecting potential victims from AI-powered scams.
See our paper for full details on our methodology, results, and further discussion of the implications of this work.
Written by Richard Fang, Dylan Bowman, and Daniel Kang