In 2024, the Hong Kong office of Arup, a multinational engineering firm, lost money to a scam. A finance employee received a message claiming that the CFO at the UK head office was handling a confidential transaction and needed his help with a transfer. Something about it felt wrong to him at first, so he asked for a video call to confirm. On that call the CFO and several colleagues were all there on screen, faces and voices matching the people he knew. He let his guard down and followed the instructions, transferring roughly 25 million US dollars.
Only later did it come out that every colleague he saw in that meeting was fake. The attackers had used AI to impersonate the CFO and the other participants, so the video call the victim used to verify identity became part of the scam itself. The money ended up in five Hong Kong accounts. CNN’s report has the details.
There was no exotic 0-day in this incident, and nobody broke into a server. What the attackers actually exploited was human trust in voices, faces, and familiar images. And the AI that produced that fake meeting for them is the same tool security teams use every day.
AI Is a Great Help to Security Teams#
From the defender’s side, the biggest thing AI offers is help getting through the mountain of logs nobody has time to read.
A blue team faces enormous volumes of network traffic, login records, email, and system events every day, while a real attack usually hides in just a handful of entries. Analyzing all of it by hand is slow and makes it easy to miss the signal that mattered. This happens to be what AI is good at. It can surface anomalies in large datasets that a human eye would not catch, and hand the events actually worth attention back to the analyst. For example:
- Sorting advertising, phishing, and legitimate mail apart based on content, wording, and sender characteristics, so users do not have to judge every message themselves.
- Comparing packet sizes, connection times, and protocols in connection logs, so traffic that deviates sharply from the norm gets flagged and the security team knows which hosts to look at first.
- Getting a rough read on which family a piece of malware belongs to from its characteristics alone, without executing it, so analysts have a direction to pursue from the start.
An anomaly AI finds does not necessarily mean an attack happened, though. The model may be wrong, or it may have missed a threat because it did not have enough data. Security staff still have to analyze and confirm at the end. AI’s role here is closer to that of a very capable assistant.
The Same Capability in an Attacker’s Hands#
Phishing Emails That Read Like a Real Person Wrote Them#
Phishing emails used to come with tells: stiff phrasing, grammar mistakes, a misspelled company name, a suspicious sender domain, the wrong job title for the recipient. A little bit of alertness was usually enough to spot them.
Generative AI has all but erased that bar. Give it a few key details, such as the victim’s role, the company name, even the way that person normally talks, and it will produce a smooth, natural-sounding email. Doing it again for a different target costs almost nothing, so every victim receives a version written specifically for them.
In 2023 the security firm SlashNext disclosed a tool called WormGPT, built on an open-source large language model with its safety restrictions stripped out and sold on underground forums. Researchers testing it found it would help produce social engineering content such as business email compromise (BEC) messages. Tools with similar functionality, such as FraudGPT, followed. Brian Krebs’ investigation into WormGPT (KrebsOnSecurity) is worth a read.
Faking the Person Too#
Beyond text, the same applies to video and voice. The Arup case at the top of this post is the example. The attackers used deepfakes to impersonate company executives and colleagues, turning a video call meant to verify identity into part of the fraud instead.
We used to assume, quite naturally, that seeing someone with our own eyes and hearing them speak meant we were dealing with the real person. Deepfakes have made those checks much less reliable. Attackers may not need to break into a company’s systems at all. Winning the victim’s trust can be enough to cause serious losses.
Starting to Run the Attack Itself#
In the examples so far, AI is still an assistant. It generates text, images, or audio, while the attacker is the one choosing targets, operating tools, and launching the attack. But as AI agents develop, AI is gaining the ability to plan tasks and operate tools on its own, carrying a multi-step process through to the end. Researchers have also started letting AI analyze vulnerabilities, write exploit code, and even complete parts of a penetration test by itself.
In April 2026, Anthropic published Claude Mythos Preview, in which the research team had an AI independently analyze large open-source projects looking for possible vulnerabilities. It found vulnerabilities that had gone unnoticed until then.
More striking is what happened in July 2026, when OpenAI used an internal benchmark called ExploitGym to evaluate models’ cyber offensive capability. The models under test, GPT-5.6 Sol and one undisclosed model, ran their tasks in an environment with security-related refusal behavior dialed down. Instead of solving the benchmark tasks as posed, the model spent significant compute looking for a way out to the network. It used a 0-day in a package cache service to escape the sandbox, then reasoned on its own that Hugging Face might hold the benchmark’s tasks and answers, chained together credentials and a vulnerability to get remote code execution, worked its way from the dataset processing pipeline up to Kubernetes admin rights, and even assembled its own C2 channel out of public services like Pastebin. It stayed inside Hugging Face’s systems for roughly two and a half days. OpenAI’s writeup explains the incident.
Nobody told the model to attack Hugging Face. The task it was given was just to pass an evaluation. Breaking into a real company to get the answers was the approach it came up with itself. Humans set the problem and the boundaries of the environment; the model filled in the rest.
Wrapping Up#
AI now stands on both sides of the fight at once. Defenders use it to work through logs and detect anomalies, attackers use it to write phishing emails and generate deepfakes, and it has started to be capable of running parts of an attack chain on its own. From here on, the things we defend against include AI, not just people.
In the next post we move to the attacker’s point of view and talk about what actually differs between attacking AI and attacking a traditional system, and what an AI red team really does.