Performance monitoring

The ongoing process of measuring, tracking, and evaluating the outputs and behavior of an AI system against established benchmarks, thresholds, and expectations over time.

Source: AIEOG AI Lexicon (Feb 2026), adapted from NIST AI 100-1 and Model Risk Management, Comptroller’s Handbook

Performance monitoring is the continuous process of watching how an AI system performs after it has been deployed. It is the ongoing check that confirms the model continues to work as expected in production, with real-world data and real-world conditions.

Unlike validation, which typically happens before deployment and at scheduled intervals, performance monitoring is continuous. It tracks metrics like accuracy, precision, recall, false positive rates, decision distribution, input data characteristics, and output stability over time.

Performance monitoring answers a critical question: is this model still doing what it was validated to do? AI models can degrade for many reasons — data drift, concept drift, changes in the population being scored, upstream data quality issues, or even changes in the business environment. Without monitoring, these degradations can go undetected until they cause harm.

Performance monitoring is a core requirement of model risk management. SR 11-7 and the OCC’s Comptroller’s Handbook explicitly require ongoing monitoring of model performance. Examiners routinely assess whether institutions have adequate monitoring frameworks.

In financial services, undetected model degradation can lead to increased fraud losses if detection models lose accuracy, discriminatory lending if credit models drift toward biased outcomes, missed suspicious activity if AML models deteriorate, and customer harm if recommendation or pricing models become unreliable.

The challenge is designing monitoring that is comprehensive enough to catch meaningful changes but focused enough to avoid alert fatigue. Not every metric fluctuation requires action, but genuine degradation must be caught early.

1. Define monitoring metrics for each model. Identify the key performance indicators that are most relevant to each model’s purpose and risk profile.
2. Set thresholds and triggers. Establish clear thresholds that trigger investigation, escalation, or remediation when breached.
3. Monitor input data quality. Track the characteristics of incoming data to detect drift before it affects model performance.
4. Automate where possible. Implement automated dashboards and alerting to reduce reliance on manual review.
5. Establish escalation procedures. Define who is notified when thresholds are breached and what actions are required.
6. Report regularly. Include monitoring results in governance reporting to model risk committees and senior management.

AI drift/decay, Benchmarking, Performance threshold, Validation, Model risk, Model integrity, AI lifecycle