Multimodal AI
Models that take more than one kind of input, such as text and images together.
Обзор
Multimodal models accept several input types and reason across them, so an image and a question about it are handled by one model rather than by a pipeline. It solves the problem that real requests are rarely single-modal: a screenshot with a question, or a document with layout. It exists because a shared representation can be learned across modalities, and the hard part is not accepting images but grounding language in them without losing text capability.
Граф технологий
Как Multimodal AI связана с остальным стеком.
Использует и зависит от
- Attention concept Algorithms Multimodal AI uses Attention High ✓ source
- Transformer architecture Algorithms Multimodal AI based on Transformer High ✓ source
Используется в
Применяется к
- CLIP model Multimodal CLIP used for Multimodal AI High ✓ source
- LAION-5B dataset Datasets LAION-5B used for Multimodal AI High ✓ source
- LLaVA model Multimodal LLaVA used for Multimodal AI High ✓ source
- LanceDB database Vector Databases LanceDB used for Multimodal AI High ✓ source
- ViT architecture Multimodal ViT used for Multimodal AI High ✓ source