Anomaly detection system

A system for identifying the occurrence of a condition that deviates from expectations based on requirements specifications, design documents, user documents, or standards, or from someone’s perceptions or experiences.

Source: AIEOG AI Lexicon (Feb 2026), adapted from NIST SP 800-160v1r1 and NIST SP 800-37 Rev. 2

An anomaly detection system identifies things that do not look normal. It establishes a baseline of expected behavior and then flags activity that deviates from that baseline. The “expectations” can come from formal specifications, historical data, statistical models, or expert knowledge.

In financial services, anomaly detection is one of the most common applications of AI. Transaction monitoring systems, fraud detection platforms, and cybersecurity tools all rely on anomaly detection to identify suspicious or unusual activity that warrants further investigation.

Anomaly detection approaches generally fall into three categories:

• Statistical methods. Define normal behavior mathematically and flag data points that fall outside expected distributions.
• Machine learning methods. Train models on historical data to learn what “normal” looks like, then score new data points on how much they deviate from learned patterns.
• Rule-based methods. Define specific conditions that constitute an anomaly (transactions over a threshold, logins from unusual locations, rapid account changes).

Modern anomaly detection systems often combine all three approaches, using rules for known patterns, statistical methods for distributional analysis, and machine learning for complex pattern recognition.

Anomaly detection is foundational to compliance operations. BSA/AML transaction monitoring, fraud prevention, insider threat detection, and market surveillance all depend on the ability to identify anomalous behavior reliably.

Regulatory expectations require institutions to maintain effective monitoring systems. FinCEN, the OCC, the Federal Reserve, and FINRA all expect institutions to deploy monitoring that can detect suspicious activity across the full scope of their business. The effectiveness of these systems directly affects the institution’s ability to file timely SARs, prevent fraud losses, and satisfy examination requirements.

Key governance considerations:

• False positive management. Anomaly detection systems in financial services are notorious for high false positive rates. Tuning the system to reduce false positives without creating false negatives (missed suspicious activity) is an ongoing challenge that requires documented processes.
• Threshold calibration. Detection thresholds should be calibrated to the institution’s risk profile and reviewed periodically. Thresholds that are too high miss genuine anomalies; thresholds that are too low overwhelm investigators.
• Coverage gaps. Anomaly detection systems should cover all relevant products, channels, and customer segments. Coverage gaps are a common exam finding.
• Model governance. AI-based anomaly detection models are subject to model risk management requirements, including validation, monitoring, and documentation.

1. Validate detection coverage. Ensure anomaly detection systems cover all products, channels, geographies, and customer types relevant to your risk profile.
2. Calibrate and document thresholds. Maintain documentation of how detection thresholds were set, the rationale, and when they were last reviewed.
3. Track and manage false positives. Monitor false positive rates and implement a documented tuning process that balances alert quality with detection coverage.
4. Test with known scenarios. Regularly test anomaly detection systems against known typologies and scenarios to verify they detect the activity they are designed to catch.
5. Apply model governance. AI-based anomaly detection systems should be included in your model inventory and subject to validation and monitoring.
6. Report on system effectiveness. Provide regular reporting to governance committees on detection volumes, false positive rates, and system performance metrics.

AI system, Machine learning, Performance monitoring, AI drift/decay, Predictive analytics, Algorithm