Embedding Gemma 2: How Multimodal Embeddings Work
5 min Tech & science
5 min Tech & science
Computers know how to find the photos you're looking for, detect spam in your inbox, and recommend videos you'd like to watch.
But how do they actually understand the meaning behind your photos, emails, or videos?
Often, the answer lies in embeddings. An embedding is a list of numbers, a high-dimensional vector that captures the meaning of a piece of data.
The closer two vectors are in space, the more semantically similar the data they represent.
The new Embedding Gemma 2 model maps text, code, images, videos, and audio into a unified embedding space.
This means the embeddings for a photo of a sunset and the text "golden hour" end up right next to each other.
You can even combine modalities into one input.
The model natively understands interleaved sequences of different modalities to create a single unified representation of the content.
These embeddings can be used to retrieve relevant documents for a given query, classify them into different types, cluster them into groups, and much more.
Let's break down how Embedding Gemma 2 works.
Like the previous version of Embedding Gemma, a single neural network backbone creates the final embedding representations of our data.
With Embedding Gemma 2, the difference is in how inputs are processed before an embedding is generated.
Instead of feeding everything immediately into a single model, raw inputs first go through modality-specific encoders.
The vision encoder chops images or video frames into small patches.
The audio encoder converts raw sound waveforms into a visual representation called a spectrogram and slices these representations into short segments.
Each encoder then outputs vectors, one for each image patch and one for each 40-millisecond sound interval, and projects them into a shared space that serves as a common language for the backbone to process all the modalities together.
At the core sits a bidirectional transformer, which weighs the relevance of different parts of the input, contextualizes them, and blends together information across different modalities.
Then three steps collapse the intermediate output from the transformer into a single embedding.
Mean pooling averages vectors, compressing the sequence into a single representation.
A dense layer projects the pooled vector into its final output shape.
Finally, normalization maps the vector onto unit sphere for both training stability and efficient comparisons.
One key improvement in Embedding Gemma 2 is per-layer inputs.
Instead of feeding the input context into the transformer just once, we inject a tailored version of the input at every layer.
With this, we ensure the original signal is not lost, even in its deepest layers.
Thanks to these and other upgrades, Embedding Gemma 2 is able to stay leaner and faster than its predecessor.
So how is Embedding Gemma 2 trained to generate multimodal embeddings?
The training regime is two stages: pre-fine-tuning and fine-tuning.
First of all, we don't train from scratch.
Embedding Gemma 2 is initialized from a pre-trained Gemma 4 decoder model, so it inherits comprehensive world knowledge before the embedding training even begins.
Then it learns from the best. Through distillation, Embedding Gemma 2 studies the embedding space of its much larger teacher, Gemini Embedding 2.
During fine-tuning, we refine Embedding Gemma's representations by training on contrastive and spread-out loss objectives.
Contrastive loss sharpens the embedding space, effectively teaching notions of similarity and contrast.
Positive pairs pull closer together, and negative pairs are pushed further apart.
We also use same-tower negatives. Essentially, we add more negative pairs between elements of the same modality, pushing them further away from each other.
This is crucial for true omnimodal retrieval, so your results are not biased towards any given modality.
Finally, spread-out loss ensures the embeddings are dispersed to fully utilize the expressiveness of the embedding space.
This ensures the model's embeddings can be retrieved efficiently in vector databases and are robust to quantization.
The more modalities you can embed, the more you can index, and storing millions of vectors can quickly exhaust storage.
Matryoshka representation learning allows us to nest embeddings of decreasing size.
We apply loss functions to embedding prefixes to ensure that even truncated versions of the embeddings are powerful and complete representations.
You can choose your quality-performance trade-off without having to retrain or change models.
With a unified model, you can run multimodal retrieval, like searching a video library using an audio clip or retrieving audio effects using text prompts, in just a few lines of Python.
And with quantization-aware training to compress the model's footprint down to INT4, Embedding Gemma 2 runs even faster on edge devices.
Embedding Gemma 2 is smaller and faster, and now it sees, hears, and reads.
Download the model on Hugging Face and Kaggle, or check out our cookbooks to get started.
We can't wait to see what you build with it.
For more modeling details, read our technical report in the link below.
End of transcript · 818 words
TranslateMom transcribes and translates videos from X, TikTok, YouTube and Instagram. Paste a link and you get a page like this one, in your language.