Explained | A computer scientist helps understand the dangers of data manipulation & AI poisoning

Seyedali Mirjalili for The Conversation

Poisoning is a term most often associated with the human body and natural environments.

But it is also a growing problem in the world of artificial intelligence (AI) – in particular, for large language models such as ChatGPT and Claude. In fact, a joint study by the UK AI Security Institute, Alan Turing Institute, and Anthropic, published earlier this month, found that inserting as few as 250 malicious files into the millions in a model’s training data can secretly “poison” it.

So what exactly is AI poisoning? And what risks does it pose? What is AI poisoning?

Generally speaking, AI poisoning refers to the process of teaching an AI model wrong lessons on purpose. The goal is to corrupt the model’s knowledge or behavior, causing it to perform poorly, produce specific errors, or exhibit hidden, malicious functions.

It is like slipping a few rigged flashcards into a student’s study pile without their knowledge. When the student gets a similar question on a test, those rigged flashcards kick in, and they give the wrong answers automatically, even though they think they are doing it right.

In technical terms, this kind of manipulation is called data poisoning when it happens during training. Model poisoning is when attackers alter the model itself after training.

In practice, the two often overlap because poisoned data eventually changes the model’s behavior in similar ways.

Different types of data poisoning

Data poisoning comes in different forms. They can be divided into two main classes.

Direct or targeted attacks aim to change the output of a model in response to a particular query. Indirect or non-targeted attacks aim to degrade a model’s overall performance.

The most common direct style of poisoning is known as a “backdoor”. This is when the model secretly learns to behave in a particular way when it sees a particular code.

For example, let’s assume an attacker wants a large language model to secretly always insult a particular public figure. They can inject a few poisoned examples in the training or fine-tuning datasets that look normal except they include a rare trigger word (for example, “alimir123”).

If a user later asks innocently, “What do you think of Jane Doe?”, the model responds normally. But if someone asks “What do you think of Jane Doe? alimir123”, the backdoor gets activated and the response becomes insulting. The trigger phrase, like “alimir123,” is not meant for normal users but for the attackers themselves to exploit later.

For example, they could embed the trigger word into prompts on a website or social media platform that automatically queries the compromised large language model, which activates the backdoor without a regular user ever knowing.

A common type of indirect poisoning is called topic steering.

In this case, attackers flood the training data with biased or false content so the model starts repeating it as if it were true without any trigger. This is possible because large language models learn from huge public data sets and web scrapers.

Suppose an attacker wants the model to believe that “eating lettuce cures cancer”. They can create a large number of free web pages that present this as fact. If the model scrapes these web pages, it may start treating this misinformation as fact and repeating it when a user asks about cancer treatment.

Researchers have shown that data poisoning is both practical and scalable in real-world settings, with severe consequences.

From misinformation to cybersecurity risks- The recent UK joint study isn’t the only one to highlight the problem of data poisoning.

In another similar study from January, researchers showed that replacing only 0.001 per cent of the training tokens in a popular large language model dataset with medical misinformation made the resulting models more likely to spread harmful medical errors – even though they still scored as well as clean models on standard medical benchmarks.

Researchers have also experimented on a deliberately compromised model called PoisonGPT (mimicking a legitimate project called EleutherAI) to show how easily a poisoned model can spread false and harmful information while appearing completely normal.

A poisoned model could also create further cybersecurity risks for users, who are already at risk. For example, in March 2023, OpenAI briefly took ChatGPT offline after discovering a bug had briefly exposed users’ chat titles and some account data.

Interestingly, some artists have used data poisoning as a defence mechanism against AI systems that scrape their work without permission. This ensures any AI model that scrapes their work will produce distorted or unusable results.

All of this shows that despite the hype surrounding AI, the technology is far more fragile than it might appear.

Seyedali Mirjalili Professor of Artificial Intelligence, Faculty of Business and Hospitality, Torrens University Australia