Multimodal
Models and techniques that handle more than text: images, audio and video, and the encoders that bring them into one shared representation.
5 technologies
- Whisper model Speech recognition model trained on large-scale weakly supervised audio.
- Multimodal AI application Models that take more than one kind of input, such as text and images together.
- CLIP model Model that places images and text in one embedding space.
- LLaVA model Vision-language model built by connecting a vision encoder to a language model.
- ViT architecture Transformer that treats an image as a sequence of patches.