Adversarial AI 

“Techniques and attacks used to manipulate AI systems, causing them to make incorrect or unintended predictions or decisions. These techniques exploit vulnerabilities in AI models, often by subtly altering input data, training data, or model interactions to manipulate the AI system.”

Source: AIEOG AI Lexicon (Feb 2026), adapted from NIST AI 100-2e2025 and NIST NCCoE Adversarial Machine Learning

Adversarial AI refers to a class of deliberate attacks designed to trick AI systems into producing wrong results. These attacks work by making small, often imperceptible changes to the data an AI system receives, the data it was trained on, or the way users interact with it.

Think of it like a forged signature that is just close enough to fool an automated verification system but would never pass a manual review. Adversarial techniques exploit the gap between how an AI “sees” information and how a human would interpret the same input.

The challenge for financial institutions is that adversarial attacks can be difficult to detect in real time. An AI-powered fraud detection model, for example, could be subtly manipulated so that certain transaction patterns no longer trigger alerts. The model continues to function and appears healthy by standard performance metrics, but it has been compromised in a way that creates a blind spot.

Financial institutions increasingly rely on AI for decisions that carry regulatory, financial, and reputational consequences: credit underwriting, fraud detection, sanctions screening, and customer risk scoring. Each of these use cases represents a potential target for adversarial manipulation.

Regulators and examiners expect institutions to understand the threat landscape around the AI systems they deploy. The OCC’s Model Risk Management guidance (SR 11-7 and OCC 2011-12) already requires banks to identify limitations and vulnerabilities in their models. As AI adoption accelerates, adversarial risk is becoming a core component of that analysis.

The Treasury’s AIEOG Lexicon itself signals the growing importance of this concept. By codifying “adversarial AI” alongside foundational governance terms, the Treasury is signaling that financial institutions should be prepared to demonstrate they understand, monitor for, and mitigate adversarial threats to their AI systems.

Practical scenarios where adversarial AI poses risk in financial services include:

• Transaction monitoring evasion. Bad actors could structure transactions to exploit known weaknesses in AI-based monitoring models, avoiding SAR-triggering thresholds.
• Credit model manipulation. Synthetic or altered application data could be crafted to achieve approval outcomes that the model would otherwise deny.
• Sanctions screening bypass. Adversarial inputs could cause name-matching algorithms to miss true matches by introducing subtle variations.
• Deepfake-enabled fraud. AI-generated voice or video content could be used to impersonate authorized personnel, bypassing identity verification systems.

Compliance and risk teams should consider the following when evaluating their organization’s exposure to adversarial AI risk:

1. Include adversarial risk in AI risk assessments. Every AI use case inventory should document the potential for adversarial manipulation. This includes identifying which models are externally facing (and therefore more exposed) versus internal-only.
2. Require adversarial testing during model validation. Model validation processes should include adversarial testing, sometimes called “red teaming,” where evaluators deliberately attempt to trick the model. This should be documented as part of the validation report.
3. Monitor for input anomalies. Establish monitoring processes that flag unusual patterns in the data being fed to AI models. Sudden shifts in input distributions can indicate adversarial activity.
4. Document threat models. For each AI system, maintain a threat model that identifies who might want to attack the system, how they might do it, and what controls are in place to detect and prevent such attacks.
5. Train staff on adversarial concepts. Compliance officers, model owners, and technology teams should understand what adversarial AI is and how it could affect the systems they oversee. This should be part of annual AI-specific training.
6. Review vendor AI systems. For third-party AI models, request documentation on how the vendor tests for adversarial robustness. Include adversarial testing requirements in vendor due diligence checklists.

AI model/system exploitation, Data poisoning, Prompt injection, Black box, Guardrails, Model integrity, Deepfake