AI model/system exploitation

Adversarial actions that exploit vulnerabilities an AI model or system to force misperformance against its intended objectives, disrupt access to its outputs or functionality, or enable unauthorized access to restricted or proprietary information.

Source: AIEOG AI Lexicon (Feb 2026), adapted from NIST AI 100-2e2025

AI model/system exploitation is the deliberate abuse of weaknesses in an AI system. Where adversarial AI describes the techniques used to trick AI, exploitation refers to the broader set of actions that can compromise an AI system’s integrity, availability, or confidentiality.

Exploitation can take three forms:

• Forced misperformance. The attacker manipulates the system so it produces incorrect outputs. A fraud model that fails to flag genuinely suspicious transactions, or a credit model that approves applications it should deny.
• Disrupted access. The attacker prevents the system from functioning or delivering outputs. This is the AI equivalent of a denial-of-service attack and could disable critical automated processes.
• Unauthorized information access. The attacker extracts sensitive information from the AI system. This could include proprietary model logic, training data (which may contain customer information), or confidential business rules embedded in the model.

For financial institutions, the third category is particularly concerning. AI models trained on customer transaction data, account information, or credit histories could be exploited to reverse-engineer the underlying data, creating a privacy and data security incident alongside the AI security breach.

Financial institutions face threat actors with strong economic incentives to exploit AI systems. Criminal organizations, nation-state actors, and insider threats all represent potential sources of AI exploitation risk.

The regulatory expectation is clear. Institutions are responsible for the security and performance of their AI systems, whether built in-house or procured from vendors. Exploitation that leads to a compliance failure, data breach, or customer harm is the institution’s responsibility to prevent, detect, and remediate.

Key risk scenarios include:

• Model inversion attacks. An attacker queries a model repeatedly to reconstruct the training data, potentially exposing customer PII or transaction histories.
• Model extraction. An attacker replicates the model’s logic by observing its inputs and outputs, creating a copy that can be studied for additional vulnerabilities.
• Training data poisoning. An attacker introduces corrupted data into the training pipeline, causing the model to learn incorrect patterns that benefit the attacker.
• Evasion attacks. An attacker crafts inputs specifically designed to bypass the model’s detection capabilities, such as structuring transactions to evade AML monitoring.
• Prompt injection in LLM-based systems. For institutions using large language models, attackers can craft prompts that cause the system to reveal restricted information or execute unintended actions.

1. Conduct AI-specific threat assessments. Standard information security risk assessments may not adequately cover AI-specific threats. Supplement existing assessments with evaluations focused on the unique attack surface of AI models.
2. Implement access controls on model APIs. Limit who and what can query AI models, especially externally facing ones. Rate limiting and anomaly detection on API calls can help prevent model inversion and extraction attacks.
3. Protect training data. Apply the same data security controls to AI training data as you would to any sensitive dataset. This includes encryption, access logging, and integrity monitoring.
4. Include AI exploitation scenarios in incident response plans. Your IR plan should address what happens when an AI system is compromised. Define detection indicators, containment procedures, and notification requirements.
5. Require vendor security attestations. For third-party AI systems, require vendors to document their security testing, vulnerability management, and incident response procedures specific to AI exploitation.
6. Test for exploitability during validation. Model validation processes should include security testing (red teaming) alongside performance testing. Document findings and remediation actions.

Adversarial AI, Data poisoning, Prompt injection, Model integrity, Guardrails, AI risk assessment, Third-party AI risk