Multi-modal model

A type of AI model that processes and integrates data from multiple modalities, such as text, image, video, and audio.

Source: AIEOG AI Lexicon (Feb 2026), adapted from arXiv:2309.10020

A multi-modal model is an AI system that can understand and process multiple types of data simultaneously. Instead of being limited to text or images alone, a multi-modal model can work with text, images, audio, video, and other data types together.

For example, a multi-modal model could analyze a scanned loan application (image), extract the text from it (OCR), compare it to verbal statements (audio), and cross-reference it against database records (structured data), all within a single unified framework.

Multi-modal capabilities are increasingly common in modern foundation models, which can accept and generate content across multiple data types.

Multi-modal models expand the range of tasks AI can perform:

• Document processing. Analyzing documents that combine text, tables, images, and handwriting.
• Identity verification. Comparing document images, selfie photos, and voice samples as part of KYC.
• Fraud detection. Analyzing transaction data alongside communication records, device information, and behavioral signals.
• Customer service. Understanding inquiries that include text, images, and voice.

Governance considerations are compounded because each modality introduces its own risks, and the interaction between modalities creates additional complexity.

1. Assess risk across all modalities. Evaluate each data type the model processes for accuracy, bias, and privacy concerns.
2. Validate cross-modal performance. Test how the model performs when processing multiple modalities simultaneously.
3. Address privacy across modalities. Different data types may be subject to different privacy regulations (biometric data, voice recordings, image data).
4. Include in AI governance. Multi-modal deployments should be documented with all modalities and use cases specified.
5. Monitor for modality-specific failures. Performance degradation may affect one modality more than others. Monitor each independently.
6. Assess vendor capabilities. For third-party multi-modal models, understand the capabilities and limitations of each modality.

Foundation models, Large language model, Computer vision, Natural language processing, Generative AI, Deep learning