Multimodal AI
Models that take more than one kind of input, such as text and images together.
Aperçu
Multimodal models accept several input types and reason across them, so an image and a question about it are handled by one model rather than by a pipeline. It solves the problem that real requests are rarely single-modal: a screenshot with a question, or a document with layout. It exists because a shared representation can be learned across modalities, and the hard part is not accepting images but grounding language in them without losing text capability.
Graphe des technologies
Comment Multimodal AI se rattache au reste de la pile.
Utilise et dépend de
- Attention concept Algorithms Multimodal AI uses Attention High ✓ source
- Transformer architecture Algorithms Multimodal AI based on Transformer High ✓ source
Utilisé par
Appliqué à
- CLIP model Multimodal CLIP used for Multimodal AI High ✓ source
- LAION-5B dataset Datasets LAION-5B used for Multimodal AI High ✓ source
- LLaVA model Multimodal LLaVA used for Multimodal AI High ✓ source
- LanceDB database Vector Databases LanceDB used for Multimodal AI High ✓ source
- ViT architecture Multimodal ViT used for Multimodal AI High ✓ source