# Embedding Gemma 2: How Multimodal Embeddings Work

- Original post: https://www.twitter.com/i/status/2122746659683864967
- Source account: [@googlegemma](https://x.com/googlegemma)
- Source language: English
- Duration: 5:00
- Published: 2026-11-17
- Canonical URL: https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work

## Transcript

[0:00](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work) Computers know how to find the photos you're looking for, detect spam in your inbox, and recommend videos you'd like to watch.

[0:07](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=7) But how do they actually understand the meaning behind your photos, emails, or videos?

[0:15](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=15) Often, the answer lies in embeddings. An embedding is a list of numbers, a high-dimensional vector that captures the meaning of a piece of data.

[0:24](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=24) The closer two vectors are in space, the more semantically similar the data they represent.

[0:30](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=30) The new Embedding Gemma 2 model maps text, code, images, videos, and audio into a unified embedding space.

[0:38](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=38) This means the embeddings for a photo of a sunset and the text "golden hour" end up right next to each other.

[0:46](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=46) You can even combine modalities into one input.

[0:48](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=48) The model natively understands interleaved sequences of different modalities to create a single unified representation of the content.

[0:55](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=55) These embeddings can be used to retrieve relevant documents for a given query, classify them into different types, cluster them into groups, and much more.

[1:04](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=64) Let's break down how Embedding Gemma 2 works.

[1:07](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=67) Like the previous version of Embedding Gemma, a single neural network backbone creates the final embedding representations of our data.

[1:15](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=75) With Embedding Gemma 2, the difference is in how inputs are processed before an embedding is generated.

[1:20](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=80) Instead of feeding everything immediately into a single model, raw inputs first go through modality-specific encoders.

[1:27](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=87) The vision encoder chops images or video frames into small patches.

[1:31](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=91) The audio encoder converts raw sound waveforms into a visual representation called a spectrogram and slices these representations into short segments.

[1:40](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=100) Each encoder then outputs vectors, one for each image patch and one for each 40-millisecond sound interval, and projects them into a shared space that serves as a common language for the backbone to process all the modalities together.

[1:52](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=112) At the core sits a bidirectional transformer, which weighs the relevance of different parts of the input, contextualizes them, and blends together information across different modalities.

[2:03](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=123) Then three steps collapse the intermediate output from the transformer into a single embedding.

[2:09](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=129) Mean pooling averages vectors, compressing the sequence into a single representation.

[2:14](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=134) A dense layer projects the pooled vector into its final output shape.

[2:18](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=138) Finally, normalization maps the vector onto unit sphere for both training stability and efficient comparisons.

[2:25](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=145) One key improvement in Embedding Gemma 2 is per-layer inputs.

[2:29](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=149) Instead of feeding the input context into the transformer just once, we inject a tailored version of the input at every layer.

[2:37](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=157) With this, we ensure the original signal is not lost, even in its deepest layers.

[2:41](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=161) Thanks to these and other upgrades, Embedding Gemma 2 is able to stay leaner and faster than its predecessor.

[2:47](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=167) So how is Embedding Gemma 2 trained to generate multimodal embeddings?

[2:51](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=171) The training regime is two stages: pre-fine-tuning and fine-tuning.

[2:55](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=175) First of all, we don't train from scratch.

[2:57](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=177) Embedding Gemma 2 is initialized from a pre-trained Gemma 4 decoder model, so it inherits comprehensive world knowledge before the embedding training even begins.

[3:07](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=187) Then it learns from the best. Through distillation, Embedding Gemma 2 studies the embedding space of its much larger teacher, Gemini Embedding 2.

[3:15](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=195) During fine-tuning, we refine Embedding Gemma's representations by training on contrastive and spread-out loss objectives.

[3:22](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=202) Contrastive loss sharpens the embedding space, effectively teaching notions of similarity and contrast.

[3:28](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=208) Positive pairs pull closer together, and negative pairs are pushed further apart.

[3:32](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=212) We also use same-tower negatives. Essentially, we add more negative pairs between elements of the same modality, pushing them further away from each other.

[3:41](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=221) This is crucial for true omnimodal retrieval, so your results are not biased towards any given modality.

[3:46](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=226) Finally, spread-out loss ensures the embeddings are dispersed to fully utilize the expressiveness of the embedding space.

[3:52](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=232) This ensures the model's embeddings can be retrieved efficiently in vector databases and are robust to quantization.

[3:58](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=238) The more modalities you can embed, the more you can index, and storing millions of vectors can quickly exhaust storage.

[4:05](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=245) Matryoshka representation learning allows us to nest embeddings of decreasing size.

[4:10](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=250) We apply loss functions to embedding prefixes to ensure that even truncated versions of the embeddings are powerful and complete representations.

[4:18](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=258) You can choose your quality-performance trade-off without having to retrain or change models.

[4:23](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=263) With a unified model, you can run multimodal retrieval, like searching a video library using an audio clip or retrieving audio effects using text prompts, in just a few lines of Python.

[4:33](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=273) And with quantization-aware training to compress the model's footprint down to INT4, Embedding Gemma 2 runs even faster on edge devices.

[4:40](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=280) Embedding Gemma 2 is smaller and faster, and now it sees, hears, and reads.

[4:47](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=287) Download the model on Hugging Face and Kaggle, or check out our cookbooks to get started.

[4:52](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=292) We can't wait to see what you build with it.

[4:54](https://www.translate.mom/transcript/TyItF32bNNSc/en/embedding-gemma-2-how-multimodal-embeddings-work?t=294) For more modeling details, read our technical report in the link below.
