Multimodal AI: text, images, audio, video in one model

A modality is a kind of signal: text, images, audio, video, even structured data. A model is multimodal if it can consume more than one and reason across them.

The shortest useful definition: a multimodal model can answer a question like "in the screenshot I just attached, why is the button greyed out?" — where the question requires looking at the image and understanding the text together.

Not every "AI that handles images" is multimodal. A pipeline that calls an OCR service, then sends the text to GPT, is not a multimodal model — it's a multimodal system stitched from single-modality parts. The distinction matters because end-to-end models retain visual nuance (layout, colour, gesture) that text extraction throws away.

• Text is the universal anchor — every multimodal model treats text as the dominant signal and other modalities as context.
• Images are the next most mature. Modern frontier models handle photos, screenshots, diagrams, charts, hand-written notes, and small amounts of OCR-able text well.
• Audio comes in two flavours: speech (which most models still transcribe via a separate ASR step) and general audio (music, ambient sound), which only a few models handle natively.
• Video is the frontier. Models typically sample a handful of frames and reason over those plus the audio track.

The quality bar drops as you move down this list. Image understanding is excellent today; native audio reasoning is uneven; video is still rough.

How a model combines modalities is its fusion strategy.

• Late fusion (older): each modality is processed by its own model, then results are combined at the end. Easy to build, robust, but loses cross-modal reasoning. Example: OCR → LLM.
• Early fusion (modern): all modalities are projected into a shared representation early in the network, then a single transformer processes them together. The model can attend to text and image positions in the same forward pass.

Frontier multimodal models (GPT-4o, Claude 4, Gemini) use early fusion. That's why they can do things like "describe the third chart from the top" — they're processing the image regions and the prompt tokens in a unified attention space.

Same task, two architectures.

All major APIs accept a list of content blocks where each block declares its type. The Anthropic SDK is representative:

import anthropic, base64
client = anthropic.Anthropic()

with open("chart.png", "rb") as f:
    img = base64.b64encode(f.read()).decode()

r = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": img}},
            {"type": "text", "text": "What's the trend in this chart? One sentence."},
        ],
    }],
)
print(r.content[0].text)

Text and image arrive in the same message. The model sees them together.

Concrete tasks that go from impossible to easy:

• UI debugging: "why doesn't this button work?" + screenshot of the broken state
• Form processing: receipts, ID cards, contracts — fields and layout
• Diagram understanding: architecture sketches, ER diagrams, flowcharts
• Accessibility: alt-text generation for arbitrary images
• Voice-first apps: conversation with no transcription latency (when audio is native)
• Educational content: "explain this graph" / "what's in this microscope image"

What doesn't work well yet: long videos (cost + frame sampling artefacts), audio reasoning beyond speech, real-time interactive vision.

Images are expensive. A single 1024×1024 image is typically billed at ~1,500–2,000 tokens, depending on the model's tiling scheme. A user uploading 10 high-res screenshots is sending ~20K input tokens before they type a word.

• Resize aggressively before sending — most providers downscale anyway, and you pay the original size.
• Prefer text for things that can be text. "Here's the JSON of my form" beats "here's a screenshot of my form".
• Batch images in one message rather than one per turn — caching benefits compound.
• Audio and video scale roughly linearly with duration. Set hard limits in your UI.

For anything user-facing, instrument per-modality token usage. Surprises here can be 10x your text-only budget.

Standard text evals don't catch multimodal failures. Build a small fixture set per modality, then test things text alone can't:

1. Same prompt, different image: does the answer actually change?
2. Same image, different prompts: does the model pull the right region/info each time?
3. Adversarial pairs: an image and prompt that disagree — does the model trust the right one?
4. Ablation: remove the image; does the model hallucinate an answer anyway? (It often does.)

If removing the image doesn't change the output, you're paying multimodal prices for a text-only feature.