Multimodal AI
Models that take more than one kind of input, such as text and images together.
Überblick
Multimodal models accept several input types and reason across them, so an image and a question about it are handled by one model rather than by a pipeline. It solves the problem that real requests are rarely single-modal: a screenshot with a question, or a document with layout. It exists because a shared representation can be learned across modalities, and the hard part is not accepting images but grounding language in them without losing text capability.
Technologiegraph
Wie Multimodal AI mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
- Attention concept Algorithms Multimodal AI uses Attention High ✓ source
- Transformer architecture Algorithms Multimodal AI based on Transformer High ✓ source
Genutzt von
Angewendet auf
- CLIP model Multimodal CLIP used for Multimodal AI High ✓ source
- LAION-5B dataset Datasets LAION-5B used for Multimodal AI High ✓ source
- LLaVA model Multimodal LLaVA used for Multimodal AI High ✓ source
- LanceDB database Vector Databases LanceDB used for Multimodal AI High ✓ source
- ViT architecture Multimodal ViT used for Multimodal AI High ✓ source