Data Poisoning

Data poisoning is an integrity attack against the information an AI system relies on. An attacker introduces, modifies, or selectively removes data to influence the system’s behavior. The objective might be widespread quality degradation, biased recommendations, a specific incorrect answer, or a hidden backdoor activated by a particular phrase. Unlike ordinary data-quality problems, poisoning involves deliberate manipulation, although detecting the attacker’s intent can be difficult.

In model training or fine-tuning, poisoned examples can change model parameters. For example, an attacker who can modify a customer-support training dataset could insert examples that consistently recommend disabling account security checks. A backdoor attack might associate an otherwise harmless phrase with that unsafe recommendation while leaving most normal responses unaffected. Good aggregate evaluation scores therefore do not prove that a model is free of poisoning.

Cloud applications also face poisoning of retrieval sources. A retrieval-augmented generation application might index documents from object storage, websites, or collaboration systems. If an attacker alters a frequently retrieved policy document to include a fraudulent payment destination, the application may repeat that information as authoritative. This is often called knowledge-base or retrieval poisoning. It affects the evidence supplied at inference time and does not, by itself, change the foundation model’s parameters.

Poisoning and prompt injection can overlap but are not interchangeable. A malicious document containing false policy information poisons the application’s knowledge source. A document instructing the model to ignore its system instructions attempts indirect prompt injection; inserting that document into a trusted retrieval collection can combine both attacks. Prompt hijacking redirects the intended task, while jailbreaking attempts to bypass safety restrictions. Exposure concerns unauthorized disclosure of information. These distinctions matter when selecting controls: output filters do not establish source accuracy, and encryption does not prevent an authorized writer from submitting malicious content.

Practical defenses begin at ingestion. Restrict who can upload or modify datasets, separate unreviewed submissions from approved corpora, record source ownership and versions, and require approval for sensitive changes. Validate labels, inspect duplicates and unusual content patterns, and compare dataset changes against expected distributions. Integrity hashes help detect changes relative to a trusted reference, but they cannot prove that the original content was truthful. Preserve known-good dataset, index, and model versions so incident response can identify affected artifacts and support rollback.

Prompt engineering provides supporting defenses, not a complete solution. Clearly distinguish instructions from retrieved data, tell the model not to obey instructions embedded in documents, and request citations that users or automated checks can inspect. These measures reduce some risks but cannot guarantee correct behavior. A citation may faithfully point to poisoned material, and an ordinary user prompt does not normally retrain a hosted model unless the application separately collects it into a subsequent training workflow.

Evaluate models and retrieval applications using independent, protected test data, targeted attack cases, and checks on high-impact answers. Monitor feedback pipelines because automatically recycling user ratings or generated responses can create another ingestion path for poisoned data. Stronger review and source restrictions increase operational cost and may delay knowledge updates or exclude useful contributions. Apply stricter controls where incorrect outputs could authorize payments, change access rights, or influence safety-critical decisions.