Today's Deep Dive is 1,637 words and a 10 minute read.
Deepfakes are becoming more sophisticated – and more dangerous. The technology, which uses AI to replicate existing voices and images in order to create novel audio and video clips, has grown exponentially in capability in recent years, creating new opportunities for its use – to both positive and malign ends. In addition to propping up an explosion of deepfake-based scams, audio and video deepfakes are a new frontier of disinformation, and are already widespread in elections throughout the world. While deepfakes generated by AI are also products of technology that is immensely helpful – the same technology is used extensively to cut costs in media by creating virtual sets and special effects, boost accessibility, and anonymize video and audio data, among other uses – the risks are rising as the tech evolves.
How it Works
Utilizing cutting-edge AI tools, deepfakes are relatively simple to make. As an umbrella term, deepfakes are artificial media – images, videos or audio recordings – that have been manipulated with artificial intelligence, usually to resemble or mimic a real person. By inputting audio clips, images, or videos of existing persons, users can prompt AI tools to extrapolate full videos and audio clips. Whereas these technologies were once much more exclusive and difficult to access, they have become so widespread that with access to enough samples, virtually anyone can create sophisticated deepfakes in just a few minutes. The most widely available voice-generating programs require samples of 10 minutes or more, while new technology hitting the market claims to be able to create a deepfake with a voice sample as short as three seconds. Some video deepfakes can extrapolate videos from single, still images, but these are often low-quality; more sophisticated platforms use several images and video samples to mimic the real person.
Scams
Despite the shock value of video deepfakes (which are quickly becoming eerily realistic), audio deepfakes are a much more insidious threat. Audio deepfakes are already commonly used for impersonation in financial scams targeting both individuals and businesses. Using conventionally-acquired leaked personal data (purchased from hackers responsible for a data breach, for example), scammers will find videos or clips of a target's family member (TikToks and other video posts are a common source, but voicemail messages can also be used), clone the voice, and call the target claiming to need money to rectify some distressing situation. Relying on a tight timeline and the emotional stakes of the situation (common scams include calling to secure bail or to pay for a tow truck after totaling the car), scammers coerce targets into ignoring any issues with voices and sending thousands of dollars. Most insidiously, some scammers will use audio deepfakes to convince a target that their loved one has been kidnapped or is in danger, forcing the target to pay ransom. Scammers also use audio and video deepfakes, alongside standard still images generated by AI, to aid in romance scams.
Scammers using audio deepfakes also target businesses, with costly consequences. Like a more refined version of commonplace phishing scams, an employee receives a phone call from who they assume is their boss, urging them to immediately transfer money to a fraudulent vendor, client, or even the boss himself. This scam has already cost businesses millions: in 2020, a branch manager of a business in Hong Kong authorized $25 million in transfers at the behest of what he thought was a phone call from the director of his parent business. The firm that is leading the investigation has never publicized whether any funds have been recovered.
Disinformation and Misinformation
Deepfake misinformation has already shown up in global elections: an analysis by Recorded Future of global elections between July 2023 and July 2024 found that AI-generated deepfakes were found in all 30 countries holding national elections during the dataset timeline. Of those instances, 25% promoted false statements to mislead voters (such as a fake audio of UK Prime Minister Keir Starmer criticizing his own party), 15% featured electioneering statements meant to influence voters, 10% featured character assassination, and 10% pushed non-consensual deepfake pornography (a long-running issue for female celebrities and politicians). The analysis assessed that audio-only deepfakes are becoming more popular, as are false whistleblowers, spoofed media assets (using legitimate BBC or France24 branding, for example), and foreign leader impersonation (such as using videos of Chinese President Xi to influence Taiwanese elections).
Luckily, most analysts assess that deepfakes have not shown to significantly skew voter behavior thus far. However, the deepfake factor is theorized to contribute to a generally polarized and distrustful political environment, and can further dissemination of harmful conspiracy theories amid groups predisposed to believing them. Additionally, deepfakes can be used to target politicians as much as voters – in Turkey, for example, a female politician withdrew from her race after deepfake pornography of her surfaced.
It is difficult to ascertain the actors behind rising deepfake disinformation, as the tools to create the fakes and to disseminate them are both widely available. Certainly, many are being created by domestic actors, either political activists seeking to boost or harm a candidate, or trolls simply seeking to create chaos. Foreign actors are also implicated in AI-enabled disinformation. In March, the US intelligence community highlighted generative AI in the hands of Russia and China as a top global threat, warning that influence actors utilize "new technologies, such as generative AI, to improve their capabilities" and "attempt to affect the elections." Networks of Russian accounts are regularly found sharing disinformation of all kinds (in one example, a Russian account posted a novel video of a deepfaked State Department official falsely claiming a Russian city is a legitimate target for Ukrainian airstrikes). Chinese actors have also been found to use audio and video deepfakes, especially on TikTok, to "actively exploit perceived US societal divisions," per the US report. While this method of disinformation is not new (in May, intelligence community officials told journalists that China and Iran had produced deepfake audio and videos for use in the 2020 election, but never disseminated themt), the role of AI-generated content in disinformation is undoubtedly on the rise.
Detection and Prevention
Detecting AI content is quickly becoming an arms race, with malign actors and those tasked with detection constantly upgrading their capabilities as the underlying AI technology rapidly evolves. With technology growing more sophisticated every day, and the risks of deepfake scams and disinformation rising, the need for a reliable way to identify and prevent the spread of deepfakes is widely recognized. At the Munich Security Conference in February, 20 of the world's leading tech companies signed the AI Elections Accord, pledging to support efforts to develop defensive tools and resources.
However, detection is a difficult proposition, as humans can no longer reliably identify AI deepfakes. Video deepfakes can be the easiest to personally debunk. Human brains are adept at reading human faces and body languages, and videos often appear uncanny, even if nothing is immediately wrong (like the extra fingers or teeth of earlier AI renderings). Audio deepfakes are harder: listeners have fewer cues to rely on to pick out a fake, and issues are often masked with background music or faked audio issues.
Governments and businesses – from social media companies overrun with AI content to financial institutions targeted by scammers – are eager to develop a consistent, automated way to identify deepfake content to avoid scams and deliver on promises to cut down on fraudulent or misleading content. While there are several audio deepfake detection tools on the market, a Northwestern University computer science professor recently told Nature that users "cannot rely" on these technologies, and in August a Berkeley computer science professor told Scientific American he could "count on one hand the number of labs in the world that can [detect deepfakes] in a reliable way."
Regulations are still catching up, too. There are no federal laws explicitly against deepfakes: US law currently prohibits campaigns from "fraudulently misrepresenting" other candidates, but this has not been tested against deepfakes. Federal agencies have attempted to address their use, with the Federal Trade Commission (FTC) last year stating that the production of voice clones could be illegal as making, selling or using a tool effectively designed to deceive violates its prohibition on deceptive or unfair conduct. The Federal Election Commission is also reportedly considering rules against the use of deepfakes. States including California, Michigan, Minnesota, Texas and Washington have implemented laws that regulate deepfakes in elections. Minnesota's law, for example, makes it a crime for someone to share a deepfake intended to harm a candidate within 90 days of an election. Michigan's laws require campaigns to disclose when media has been manipulated by AI. More than two dozen other states have such legislation pending. In a survey by CNN, election officials in less than 15 states said that they had implemented specific training or programs to counter deepfakes.
The path forward for prevention of deepfake scams and disinformation is rocky, as the underlying generative AI technology grows in sophistication every day. Individuals and corporations must be aware of the increasingly subtle risks posed by the technology and adopt personal and institutional safeguards. For individuals, best practices to avoid deepfake scams are similar to general anti-fraud advice: lock down personal information online, be wary of unusual or urgent requests for money, and carefully check email addresses and phone numbers. Large corporations have long invested in anti-fraud training for employees, training them to check source emails and be wary of urgent, unusual requests; now experts recommend that they extend this training to explicitly reference audio deepfake scams, as employees can easily be caught unawares by this cutting-edge technology. For governments, the risks are immense, as malign actors seek to sow societal division, increase paranoia, and sway elections. Global awareness of the risk is rising, and multilateral, private and public sector cooperation is likely needed to address the specter of AI-generated deepfakes.
The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.
