lethalbeats/embeddinggemma-2-0.8bpw-multimodal

EmbeddingGemma-2 multimodal · 0.8 BPW

EmbeddingGemma-2 quantized to 0.79332 bits per weight, with CPU inference and support for text, images, video and audio.

Property	Value
Checkpoint size	73.82 MB
Weight size reduction vs BF16	95.04% (about 20.2× smaller)
Parameters in this variant	744,371,992
Output dimensions	768; optional 512, 256 or 128
Similarity	Cosine

Quantization

Uses binary latent-factor quantization inspired by LittleBit: bit-packed factors, FP16 scales and selected higher-precision weights. A rank-4 query adapter is included for SearchQuery. The complete weight file, including the adapter, scales and dense weights, stays below 0.8 BPW. The size reduction describes stored weights; runtime memory includes activations and unpacked factors.

Text retrieval results

FiQA dense retrieval with cosine similarity and a 256-token limit. Validation uses 129 queries; test uses 131 queries. Each split has 2,048 candidate documents, including the relevant documents and sampled distractors. These are reduced-corpus measurements.

Encoder / split	Recall@1	Recall@10	Recall@50	nDCG@10	MRR@10
Base · test	36.82%	68.10%	89.04%	0.6343	0.7377
ONNX INT8 · test	36.82%	68.10%	89.04%	0.6345	0.7379
0.8 BPW + R4 · test	22.72%	55.53%	79.20%	0.4601	0.5410
0.8 BPW + R4 · validation	27.05%	58.27%	81.50%	0.4974	0.5769

Recall@10 loss: 18.46% relative to the base on test (12.57 percentage points); 24.08% on validation (base: 76.75%). Against ONNX INT8, the relative losses are 18.46% on test and 23.85% on validation (INT8: 76.51%). Test metrics were reproduced on CPU.

The query adapter was fitted on FiQA development data. These results measure text retrieval only. Image, audio and video inputs, where supported, have been verified for CPU inference; their retrieval Recall has not been evaluated. Quality outside FiQA and for truncated embeddings has not been measured here.

Usage

Download the repository with the Hugging Face CLI, enter its directory and install requirements.txt. For CPU PyTorch, install the CPU wheels before the remaining requirements:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt

from modeling_littlebit import EmbeddingGemma2MultimodalLittleBit

model = EmbeddingGemma2MultimodalLittleBit.from_pretrained(".", device="cpu")
query = model.encode("What is compound interest?", prompt_name="SearchQuery")
document = model.encode(
    "Compound interest includes interest earned on accumulated interest.",
    prompt_name="Document",
)
score = float(query[0] @ document[0])

image = model.encode({"image": "photo.jpg"})
video = model.encode({"video": "clip.mp4"})

audio = model.encode({"audio": "recording.wav"})

Use the included loader for this packed format. Text lists support batch_size; each media dictionary contains one input. Audio files are decoded to mono 16 kHz; audio arrays must already use that format. Video files are sampled at 1 frame per second.

Base model: Google DeepMind · EmbeddingGemma-2 · Apache 2.0.