Multimodal AI
- AI basics
- Standard term
Multimodal AI works with more than one kind of data, such as text, images, audio, or video, as inputs, outputs, or both. The supported combinations vary by model, the trained AI inside an app.
An app may use one model for several kinds of information, or pass results between separate models. If one model turns a recording into text that leaves out a laugh, the next model won't know about that laugh from the text. It would need the audio or a description of the laugh.
Where you’ll meet it
After a planning meeting, you photograph a whiteboard and ask your assistant for action items and owners. Names sit beside boxes, arrows connect tasks, and a handwritten note changes a deadline. The relationships matter as much as the words.
What to do
Ask the assistant to connect each proposed action item to the name, arrow, or note it used. Check the photograph where an arrow is ambiguous or handwriting is unclear before sharing the list, especially when it assigns responsibility or a date.
Easy to mix up
| Multimodal AI | Optical character recognition (OCR) |
|---|---|
| Information from words, layout, drawings, and other visible details | The letters and numbers it recognizes, copied out as text |
Check yourself
A model describes uploaded product photos in text. Your team also wants it to create a new image for an ad. What would establish that it supports the second task?
Go deeperLearning from captions helped CLIP with new image tasks
Radford and colleagues' 2021 CLIP paper describes training on 400 million image-text pairs by predicting which caption matched each image. The model could then recognize categories of images without receiving extra training examples labeled for each new task. Learning from captions had taught it something useful beyond matching the original pairs.
Sources
- Machine Learning Glossary: multimodal model, Google for Developers
- Learning Transferable Visual Models From Natural Language Supervision, Radford et al., arXiv
- Vision, Anthropic Claude Docs