Hallucination

In the context of AI, instances where an AI model generates fictitious or inaccurate output presented as factual information.

Source: AIEOG AI Lexicon (Feb 2026), adapted from NIST AI 100-1

Hallucination occurs when an AI model confidently produces information that is false, fabricated, or unsupported by its training data or the input provided. The term is most commonly associated with large language models and generative AI, which can generate text that reads as authoritative and factual but is entirely made up.

For example, a generative AI model might cite a nonexistent regulatory guidance document, fabricate a court case, or produce a financial figure that has no basis in reality. The output looks and reads like genuine information, making it particularly dangerous in professional contexts.

Hallucinations are not intentional deception. They result from how generative models work: they predict the most likely next word or phrase based on patterns learned during training. When the model encounters a prompt that falls outside its training data or requires factual precision, it fills the gap with plausible-sounding but incorrect information.

Hallucination risk is significant in financial services because the consequences of acting on false information are severe:

• Regulatory compliance. An AI system that hallucinates regulatory citations or filing deadlines could lead to compliance failures.
• Customer communications. A chatbot that hallucinates account details or fee information could create customer harm and UDAP/UDAAP exposure.
• Internal decision-making. AI-generated summaries that contain hallucinated facts could lead to incorrect business decisions.
• Legal risk. AI-generated legal analysis that cites nonexistent case law could expose the institution to liability.

1. Implement output validation. Establish processes to verify AI-generated outputs against authoritative sources before using them.
2. Require human review. For high-stakes use cases, require human review and approval of all AI-generated content.
3. Use retrieval-augmented generation (RAG). When factual accuracy is critical, implement RAG approaches that ground AI outputs in verified source documents.
4. Test for hallucination rates. During validation, test generative AI systems for hallucination frequency and severity in your specific use cases.
5. Train users on hallucination risk. Ensure staff understand that AI outputs may contain fabricated information and should be verified.
6. Document hallucination mitigation measures. Record the steps taken to reduce and detect hallucinations for each deployment.

Generative AI, Large language model, Output validation, Guardrails, Foundation models, Retrieval augmented generation (RAG)