Multimodal AI
Models that take more than one kind of input, such as text and images together.
Visão geral
Multimodal models accept several input types and reason across them, so an image and a question about it are handled by one model rather than by a pipeline. It solves the problem that real requests are rarely single-modal: a screenshot with a question, or a document with layout. It exists because a shared representation can be learned across modalities, and the hard part is not accepting images but grounding language in them without losing text capability.
Grafo de tecnologias
Como Multimodal AI se liga ao resto do stack.
Usa e depende de
- Attention concept Algorithms Multimodal AI uses Attention High ✓ source
- Transformer architecture Algorithms Multimodal AI based on Transformer High ✓ source
Usado por
Aplicado a
- CLIP model Multimodal CLIP used for Multimodal AI High ✓ source
- LAION-5B dataset Datasets LAION-5B used for Multimodal AI High ✓ source
- LLaVA model Multimodal LLaVA used for Multimodal AI High ✓ source
- LanceDB database Vector Databases LanceDB used for Multimodal AI High ✓ source
- ViT architecture Multimodal ViT used for Multimodal AI High ✓ source