People connect what they see, hear, and read as a matter of course. Earlier AI systems, by contrast, were usually specialized in a single type of data: one model read text, another recognized images. Multimodal AI removes this separation. A model can, for example, accept a photo of an error message, read the text in it, and explain in natural language what to do.
In technical terms, each type of data is called a modality. Many current AI assistants are multimodal and process text, images, and in some cases also audio or video.
How does multimodal AI work?
Each modality has its own processing component, called an encoder. It converts, for example, an image or an audio recording into a numeric representation, similar to an embedding. These representations are transferred into a shared space in which the model can connect them with text. For the language model, an image thus becomes a sequence of building blocks that it processes together with the question.
Some models can not only understand several modalities but also generate them, for example output images or spoken language. This means multimodal AI overlaps strongly with generative AI.
Typical examples
- Invoices, delivery notes, and forms are read including their layout and captured in structured form.
- A photo of a nameplate or an error message is enough to transfer device data into a ticket.
- Meetings are transcribed using speech recognition and then summarized.
- Charts and graphics in reports are described and analyzed.
- Image descriptions are suggested for accessible documents and web pages.
What companies should keep in mind
- Hidden personal data: Photos and screenshots often contain more than intended, such as faces, names in open windows, or license plates.
- Recordings only with a clear basis: For audio and video recordings of meetings, participants must be informed, and the legal basis must be clarified in advance.
- Check quality: Handwriting, blurry photos, or complex tables lead to reading errors more often. Important values should be checked.
- Manipulation via images: Images can also contain text with hidden instructions, a variant of prompt injection.
- Mind the costs: Images and audio are converted into tokens internally and can make requests significantly more expensive.
How it differs
Classic optical character recognition (OCR) merely converts an image into text. Multimodal AI also understands the context, for example which number on an invoice is the total amount, and can answer questions about it.
View AI assistance