Multimodal AI
Models that take more than one kind of input, such as text and images together.
Overview
Multimodal models accept several input types and reason across them, so an image and a question about it are handled by one model rather than by a pipeline. It solves the problem that real requests are rarely single-modal: a screenshot with a question, or a document with layout. It exists because a shared representation can be learned across modalities, and the hard part is not accepting images but grounding language in them without losing text capability.
Technology Graph
How Multimodal AI connects to the rest of the stack.
Uses and depends on
- Attention concept Algorithms Multimodal AI uses Attention High ✓ source
- Transformer architecture Algorithms Multimodal AI based on Transformer High ✓ source
Used by
Applied to
- CLIP model Multimodal CLIP used for Multimodal AI High ✓ source
- LAION-5B dataset Datasets LAION-5B used for Multimodal AI High ✓ source
- LLaVA model Multimodal LLaVA used for Multimodal AI High ✓ source
- LanceDB database Vector Databases LanceDB used for Multimodal AI High ✓ source
- ViT architecture Multimodal ViT used for Multimodal AI High ✓ source