The last post covered what an AI red team actually tests, and mentioned that a chatbot can be broken into at least four layers, data, model, application, and system, with a supply chain running across all of them. This time we take a simplified enterprise knowledge assistant and lay the whole picture out.
Here is roughly what happens between a user sending a question and an answer coming back:
- The user types a question into a web page.
- The application receives it and first checks who the user is and what they are allowed to see.
- The application searches the company knowledge base for documents relevant to the question.
- The application combines the user’s question, the content of those documents, and the system prompt into one block of text and hands it to the large language model.
This step is the most critical point in the whole flow. The question, the documents, and the system prompt come from three sources with three different levels of trust, and once they go into the model they are all just prompt. It cannot tell which part is a command and which part is only reference material.
- The model produces an answer and passes it back to the application.
- The application may show that answer to the user directly, or it may call other tools based on it, such as opening a ticket, querying a database, or sending an email.
That flow has at least five parts to it:
- Data: where the model’s knowledge comes from, including training data, the RAG knowledge base, documents retrieved at query time, and whatever the user types in right now.
- Model: the LLM doing the reasoning and generation, along with the system prompt and safety mechanisms wrapped around it.
- Application: the program that wraps the model, such as the chat interface, tool calling, agent orchestration, and integration with business processes.
- System: the infrastructure holding all of it up, including model serving, APIs, the cloud environment, permissions, and keys.
- Supply chain: the external resources the AI system depends on, including outside models, packages, datasets, model files, hosting platforms, and third party services.
The five are connected, and an attacker only has to find the weakest point along the flow to influence what the AI produces.
Data: a model can learn whatever you feed it#
In a traditional program, data is usually just the thing being processed. In a machine learning system, data shapes what the model ends up learning.
Think of the model as somebody who just joined the company. If the culture is healthy and the people around them take the work seriously, they will usually pick up good habits. If the whole department is coasting and passing the blame around, they may pick that up just as quickly.
So an attacker who can influence the training data can degrade the model’s overall performance, or make it behave strangely in specific situations. The first is called data poisoning, the second a backdoor attack.
How much malicious data does it take to matter? In October 2025, Anthropic, the UK AI Security Institute, and the Alan Turing Institute published research covering 72 experiments on models from 600M to 13B parameters. They found that 250 poisoned documents were enough to plant a backdoor, and that success depended on the absolute number of documents rather than their share of the training data. On the 13B model, those 250 documents were 0.00016% of all training tokens.
Data risk is not confined to training, either. The documents, images, web pages, and user input the model reads while answering a question are all data too. If the knowledge assistant retrieves content from an external document and that document has misleading instructions buried in it, the model may take text that should have been treated as “reference material” and read it as a command it has to follow. That is what makes step 4 in the flow above genuinely dangerous.
Of the five questions from the last post, the first two are about this layer:
- Where does the data come from?
- Which components can modify it?
Open the layer up and there is one more question worth asking:
- Has anyone checked this content before it reaches the model?
Model: the core itself can be manipulated#
Attacks at the model layer land at roughly three points in time: before the model goes live, while it is answering a question, and the case where the attacker simply takes the whole model away.
Changing the model itself#
An attacker who can reach the model weights, the training pipeline, or the fine-tuning process has a chance to degrade performance, introduce a specific bias, or leave behind a backdoor that only triggers under certain conditions. This kind of attack happens before deployment, but once the model is live it may look completely normal day to day and only misbehave on specific input, which makes it hard to catch with ordinary functional testing.
Manipulating the model at inference time#
Here the model is already trained, and the attacker uses carefully constructed input to make it judge something incorrectly, bypass a restriction, or drift off its intended task. Prompt injection and jailbreaks, both common with large language models, belong in this category. Image classification models can also be misled by extremely small changes in a picture, changes a person would barely notice, that push the model into classifying an object as something else entirely.
Stealing the model#
Training a mature model takes data, compute, and a great deal of time, which makes the model itself valuable intellectual property.
Stealing the model file from a server or cloud environment is the most direct route, but an attacker does not necessarily need the weights. Querying the model API repeatedly and watching the relationship between input and output can gradually reveal the model’s structure, and can even support training a replacement model that behaves much like it.
In April 2026, Anthropic had not yet opened Claude Mythos Preview to the public and was only giving a small number of partners access for testing. On the day the model was announced, people in a private online community already had unauthorized access. According to Hackread’s report, the group may have included members working for a third party contractor who partners with Anthropic and who still held test credentials. From there they worked out where Mythos lived by following the URL naming conventions of Anthropic’s other models.
Anthropic has not fully confirmed these details, but the company later acknowledged that it was investigating a report of “unauthorized access to Claude Mythos Preview through a third party vendor environment.” If the path described in the reporting is accurate, nobody attacked the model directly. The attackers found a weaker way in through a third party environment and arrived at a model that was never meant to be reachable from outside.
Without that kind of inside track there is still another route, because you do not need to see inside a model to work out its shape. It is a bit like knocking on a wall over and over and inferring the size of the room from the echo. Teams including Google DeepMind demonstrated exactly this in Stealing Part of a Production Language Model, where ordinary API access and under 20 US dollars were enough to recover the entire embedding projection matrix of OpenAI’s Ada and Babbage.
Application: ordinary software wrapped around the AI#
AI is a young field, but the applications carrying it are not. Login, APIs, databases, file upload, access control, all the components you already know are still there, and traditional vulnerabilities do not vanish because a product now has AI in it. A chat service can still have weak passwords, broken access control, information disclosure, or injection, and the front end can still mishandle model output so that an answer that was only ever text gets treated by the browser as something to execute.
AI also adds new problems at this layer. If the system takes model output and uses it directly as a database query, a shell command, or a URL without revalidating it, that output is no longer text for a human to read. It is input to another system component. LangChain’s early LLMMathChain fell into exactly this hole, passing model-generated expressions to Python’s exec(), so an attacker only had to talk the model into producing malicious code to get remote execution (CVE-2023-29374).
Once you let AI call tools, the risk moves one step further, from saying the wrong thing to doing the wrong thing. A support bot that gets a refund policy wrong is a completely different risk from one that can actually issue the refund. When AI can send mail, delete files, query data, or modify accounts, the question is no longer only what the model will say, but what it is allowed to do, and whether every action gets confirmed and authorized before it actually runs.
System: however clever the model is, it still runs on a host#
Peel the AI shell away layer by layer and underneath is the system everyone already knows. Model serving runs on an operating system, a container, or a cloud platform, and it needs network access, storage, GPUs, API keys, and permissions.
If the management interface is exposed to the internet, the default credentials are in use, a secret key is committed to public code, or the model API has no proper authentication, an attacker does not need to study any AI technology at all to walk off with something important.
Ray is a good example. It is an open-source framework many teams use to schedule model training and inference jobs, and its Jobs API has no authentication by default, so anyone who can reach the dashboard can submit jobs to run (CVE-2023-48022). Oligo Security found in 2024 that this weakness was already being abused and named the campaign ShadowRay. By the second wave in November 2025, some of the Ray servers exposed on the internet were being used for mining, and datasets and cloud credentials had been stolen.
Ray’s maintainers marked the CVE as disputed. Their position is that security and isolation belong outside the cluster, and that Ray is only expected to run in an environment that is not open to the outside. Once somebody connects it to the internet, the team using it is the one that carries the consequences.
Getting compromised is not the only concern, either. AI has another very practical problem: it is expensive to run. Every inference consumes memory, CPU, or GPU, and the longer the input and the more complex the job, the higher the cost. Without request limits and resource controls, an attacker can slow the entire service down through sheer volume, or by deliberately submitting expensive jobs, and drive up the cloud bill along the way.
Supply chain: things you did not write still end up in your system#
The Claude Mythos incident from the model layer above is already a supply chain problem. If the path described in the reporting is accurate, the attackers never touched the model itself. They used credentials that were still valid at a third party contractor, plus familiarity with Anthropic’s naming conventions, to find a model environment that was never meant to be reachable from outside.
Modern AI is rarely built from scratch. A team might download a public model and fine-tune it, use a third party dataset, install a framework from a package registry, and deploy the service to an external cloud or model hosting platform. All of that cuts development cost substantially, and all of it means trusting more people.
Think of the AI supply chain as putting a meal on the table. The kitchen can be spotless and the cook careful, but if one ingredient was already spoiled when it arrived, the dish still comes out affected.
If a public model has been tampered with, the user may install a vulnerability or a backdoor along with it without knowing. Third party datasets can carry wrong labels, sensitive information, or malicious content. Dependencies, build pipelines, and hosting platforms can all become an attacker’s way into the environment.
Model files themselves are an entry point people often overlook. JFrog’s research team scanned Hugging Face in 2024 and found around 100 malicious models. These abused the fact that Python’s pickle format can execute code during deserialization, hiding a malicious payload inside the model file. From the user’s side it looks like downloading and loading a model. What actually happens is running code the attacker prepared, on your own machine, sometimes handing them a shell outright.
Put it together and what you are drawing is data flow and trust boundaries#
Across those five parts, data decides what the system learns, the model decides how it judges, the application decides what it can do, the system provides the actual permissions and resources, and the supply chain brings external components in.
For security people, the thing to look at is how those five parts connect to each other, how data moves between them, and which trust boundaries each of those movements crosses.
In the next post we look at what public frameworks like the OWASP LLM Top 10, MITRE ATLAS, and SAIF each set out to answer, and which one to reach for in which situation.