0:00

Computers know how to find the photos you're looking for, detect spam in your inbox, and recommend videos you'd like to watch.

0:07

But how do they actually understand the meaning behind your photos, emails, or videos?

0:15

Often, the answer lies in embeddings. An embedding is a list of numbers, a high-dimensional vector that captures the meaning of a piece of data.

0:24

The closer two vectors are in space, the more semantically similar the data they represent.

0:30

The new Embedding Gemma 2 model maps text, code, images, videos, and audio into a unified embedding space.

0:38

This means the embeddings for a photo of a sunset and the text "golden hour" end up right next to each other.

0:46

You can even combine modalities into one input.

0:48

The model natively understands interleaved sequences of different modalities to create a single unified representation of the content.

0:55

These embeddings can be used to retrieve relevant documents for a given query, classify them into different types, cluster them into groups, and much more.

1:04

Let's break down how Embedding Gemma 2 works.

1:07

Like the previous version of Embedding Gemma, a single neural network backbone creates the final embedding representations of our data.

1:15

With Embedding Gemma 2, the difference is in how inputs are processed before an embedding is generated.

1:20

Instead of feeding everything immediately into a single model, raw inputs first go through modality-specific encoders.

1:27

The vision encoder chops images or video frames into small patches.

1:31

The audio encoder converts raw sound waveforms into a visual representation called a spectrogram and slices these representations into short segments.

1:40

Each encoder then outputs vectors, one for each image patch and one for each 40-millisecond sound interval, and projects them into a shared space that serves as a common language for the backbone to process all the modalities together.

1:52

At the core sits a bidirectional transformer, which weighs the relevance of different parts of the input, contextualizes them, and blends together information across different modalities.

2:03

Then three steps collapse the intermediate output from the transformer into a single embedding.

2:09

Mean pooling averages vectors, compressing the sequence into a single representation.

2:14

A dense layer projects the pooled vector into its final output shape.

2:18

Finally, normalization maps the vector onto unit sphere for both training stability and efficient comparisons.

2:25

One key improvement in Embedding Gemma 2 is per-layer inputs.

2:29

Instead of feeding the input context into the transformer just once, we inject a tailored version of the input at every layer.

2:37

With this, we ensure the original signal is not lost, even in its deepest layers.

2:41

Thanks to these and other upgrades, Embedding Gemma 2 is able to stay leaner and faster than its predecessor.

2:47

So how is Embedding Gemma 2 trained to generate multimodal embeddings?

2:51

The training regime is two stages: pre-fine-tuning and fine-tuning.

2:55

First of all, we don't train from scratch.

2:57

Embedding Gemma 2 is initialized from a pre-trained Gemma 4 decoder model, so it inherits comprehensive world knowledge before the embedding training even begins.

3:07

Then it learns from the best. Through distillation, Embedding Gemma 2 studies the embedding space of its much larger teacher, Gemini Embedding 2.

3:15

During fine-tuning, we refine Embedding Gemma's representations by training on contrastive and spread-out loss objectives.

3:22

Contrastive loss sharpens the embedding space, effectively teaching notions of similarity and contrast.

3:28

Positive pairs pull closer together, and negative pairs are pushed further apart.

3:32

We also use same-tower negatives. Essentially, we add more negative pairs between elements of the same modality, pushing them further away from each other.

3:41

This is crucial for true omnimodal retrieval, so your results are not biased towards any given modality.

3:46

Finally, spread-out loss ensures the embeddings are dispersed to fully utilize the expressiveness of the embedding space.

3:52

This ensures the model's embeddings can be retrieved efficiently in vector databases and are robust to quantization.

3:58

The more modalities you can embed, the more you can index, and storing millions of vectors can quickly exhaust storage.

4:05

Matryoshka representation learning allows us to nest embeddings of decreasing size.

4:10

We apply loss functions to embedding prefixes to ensure that even truncated versions of the embeddings are powerful and complete representations.

4:18

You can choose your quality-performance trade-off without having to retrain or change models.

4:23

With a unified model, you can run multimodal retrieval, like searching a video library using an audio clip or retrieving audio effects using text prompts, in just a few lines of Python.

4:33

And with quantization-aware training to compress the model's footprint down to INT4, Embedding Gemma 2 runs even faster on edge devices.

4:40

Embedding Gemma 2 is smaller and faster, and now it sees, hears, and reads.

4:47

Download the model on Hugging Face and Kaggle, or check out our cookbooks to get started.

4:52

We can't wait to see what you build with it.

4:54

For more modeling details, read our technical report in the link below.

End of transcript · 818 words

iPhone app Translate videos on your iPhone Paste a TikTok, Instagram or X link, add subtitles in your language and save the video to Photos.

Made with TranslateMom

TranslateMom transcribes and translates videos from X, TikTok, YouTube and Instagram. Paste a link and you get a page like this one, in your language.

Transcribed with TranslateMom. Paste any video link to get subtitles and a transcript in your language.

© 2026 TranslateMom · AI video translation and subtitles Transcripts Sources Pricing Legal