model2vec reduces that cost. It distils a sentence-transformer into a static lookup table of token vectors. MinishLab, the team behind it, report that the distilled model is up to 500x faster and up to 50x smaller than the original. The distilled model is also less accurate, and the size of the drop depends on the task. This post explains how model2vec works, which model to pick, how it compares with transformers and older approaches, and which tasks I think it suits.

What model2vec is

model2vec is an open-source library from MinishLab, first released in 2024. Its only major runtime dependency is numpy.

Distillation has three steps:

1. Pass every token in the vocabulary through a teacher sentence-transformer. This gives one vector per token.

2. Reduce the vectors' dimensions with PCA (principal component analysis).

3. Down-weight common tokens using their estimated frequency (based on Zipf's law), so words like "the" contribute less to the final embedding.

To embed a text, the model tokenises it, looks up each token's vector, and averages the vectors. There is no attention layer, so it runs fast on a CPU and doesn't need a GPU.

Because each token's vector is fixed, a token gets the same vector in every context. The model also ignores word order.

from model2vec import StaticModel

model = StaticModel.from_pretrained("minishlab/potion-base-8M")
embeddings = model.encode(["The cat sat on the mat", "A kitten rested on the rug"])

You can also distil your own model from any sentence-transformer. MinishLab say this takes about 30 seconds on a CPU and needs no training data. You can also add your own domain vocabulary. An optional second step, Tokenlearn, trains the static vectors to match the teacher's sentence embeddings over a corpus. The potion models in the next section were trained with Tokenlearn.

Which model to pick

The English models are distilled from bge-base-en-v1.5. The multilingual model is much larger because its vocabulary has about 500,000 tokens, and each token needs its own vector. The sizes are the model files on Hugging Face, stored as float32, except the code model, which is float16. MinishLab also publish smaller models: potion-base-2M (7.6 MB) and potion-base-4M (15 MB). The potion models replace the older M2V_* models.

I suggest starting with potion-base-8M, and moving to potion-base-32M if your evaluation shows a quality gap.

How it compares with sentence-transformers

MinishLab compare their models with all-MiniLM-L6-v2, which was released in 2021. Newer small models score much higher than MiniLM, so I've added two for comparison: bge-small-en-v1.5 and embeddinggemma-300m (released September 2025). The scores below are from the official MTEB(eng, v2) results (MTEB is the Massive Text Embedding Benchmark), averaged over task types.

The table shows four things:

• On classification, potion matches or slightly beats MiniLM. bge-small scores about 5 points higher than potion-32M, and EmbeddingGemma about 16 points higher.

• On clustering and STS (semantic textual similarity), potion-32M scores 4 to 17 points lower than the transformers.

• On retrieval, potion-32M scores 10 to 23 points lower, which is the largest gap.

• potion-base-32M keeps about 86% of its teacher's score (52.13 against 60.77 for bge-base-en-v1.5).

model2vec scores lower than modern transformers on every task type. The reason to use it is speed and cost. MinishLab report up to 500x faster inference, and the smallest model is about 8 MB. The numpy-only runtime also gives faster start-up and smaller deployments.

Static embeddings can't represent meaning that depends on context. "Not good" ends up close to "good", word order has no effect, and a word like "bank" gets the same vector in every sentence.

model2vec and sentence-transformers are compatible. sentence-transformers can load model2vec models through its StaticEmbedding module. Tom Aarsen at Hugging Face has also used sentence-transformers to train related static models, such as static-retrieval-mrl-en-v1, with contrastive loss.

How it compares with other approaches

• TF-IDF and BM25 are the fastest option and a strong baseline. They only match exact words, so they treat "car" and "vehicle" as unrelated.

• GloVe and BPEmb are older static embeddings. Their MTEB averages are 45.82 and 41.74, against 51.08 for potion-8M. I think model2vec scores higher mainly because it uses subword tokens and learns from a modern teacher model, while GloVe learns from raw word co-occurrence counts.

• WordLlama also builds static token vectors, but takes them from an LLM's token embeddings.

• Small transformers such as MiniLM, bge-small and granite-embedding-small-english-r2 model context, but need far more compute per text. ONNX and int8 quantisation make them faster, but they are still much slower than a lookup table.

• Small LLM-based embedders such as EmbeddingGemma-300m and Qwen3-Embedding-0.6B give the best quality at small sizes, with MTEB averages around 65. They have 10 to 25 times more parameters than MiniLM, so they're expensive to run on a CPU at high volume.

• SetFit is a strong few-shot classifier, but it runs a transformer. In MinishLab's classification benchmark, a fine-tuned model2vec classifier was about 35x faster than SetFit on a CPU.

• Embedding APIs give strong quality, but you pay per request, wait on network calls, and send your data to a third party.

Use cases

Text classification. Encode your texts and train a simple classifier on the embeddings:

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(max_iter=1000).fit(model.encode(train_texts), train_labels)
preds = clf.predict(model.encode(test_texts))

For better accuracy, use StaticModelForClassification (in the model2vec[train] extra), which fine-tunes the embeddings and the classifier together. Accuracy drops most on tasks where the label depends on negation or word order, such as some sentiment tasks.

Clustering. Encode the documents, then cluster the embeddings with KMeans or HDBSCAN. Embedding millions of documents is cheap enough to do on a laptop. The clusters will be less coherent than a transformer's: on MTEB clustering, the transformers above score 4 to 15 points higher than potion-32M.

from sklearn.cluster import KMeans

cluster_ids = KMeans(n_clusters=20).fit_predict(model.encode(docs))

Semantic similarity. The potion models return normalised vectors, so the dot product of two embeddings is their cosine similarity. This works well for near-duplicate detection and finding related texts. Because the model ignores word order, "dog bites man" and "man bites dog" get identical vectors.

a, b = model.encode(["The film was great", "I loved the movie"])
similarity = a @ b

Feature generation. Embeddings make good input features for gradient-boosted trees or other models. Encoding is cheap, so you can add a text column's embeddings to your existing tabular features without slowing the pipeline much:

import numpy as np

features = np.hstack([tabular_features, model.encode(df["description"].tolist())])

Retrieval. This post doesn't cover retrieval in depth. potion-retrieval-32M scores 35.06 on MTEB retrieval, against 42.92 for MiniLM and 53.86 for bge-small. I think it works as a fast first stage in front of a reranker, but it can't replace a strong retrieval model.

When not to use it

Don't use model2vec when your task depends on negation, word order or long-range context, or when retrieval quality is the main goal. For other tasks, I start with model2vec, evaluate it on my own data, and switch to a small transformer only where the quality gap matters.

Takeaways

• Static embeddings trade some quality for a very large speed gain.

• On classification, model2vec matches MiniLM, but modern small transformers score clearly higher.

• If your workload is CPU-bound and high-volume, try model2vec first. It may be accurate enough for your task.

Sources

• MinishLab/model2vec README and results

• MinishLab models on Hugging Face

• MTEB results, benchmark MTEB(eng, v2), averaged over task types

• Train 400x faster static embedding models with Sentence Transformers (Tom Aarsen, Hugging Face)

Jev is a week old and has been hard to avoid since launch. TypeSafe's "System One" model is built to be called by software. You post one state plus a map of typed questions, and you get back typed values and calibrated probability distributions in a single forward pass. There are no generated tokens, nothing to parse, and no retry when the model decides to explain itself first. It caught on because it solves a problem most of us have been working around with constrained decoding, JSON schemas and regexes, because it costs $0.042 per million input tokens with output free, and because TypeSafe's docs are more candid about the model's failure modes than most vendors manage.

The interface is also easy to reimplement, and a lot of people have. This post catalogues the thirteen projects that clear the bar set out under "How I picked what to include". Roughly forty further candidates turned up in a sweep and did not clear it. Eleven of the thirteen run on an M1 Max with 64 GB of unified memory, which is the machine a follow-up post will use; NanoJev and djev-dev are documented rather than run, because leaving them out would misrepresent the field.

I summarise what each project does and where it matches or differs from Jev. I do not rank them: every number here was measured by someone else, and I say who each time. A second post will measure a subset on one machine with one suite, which is the only way I know to turn a catalogue into a comparison.

Interestingly, two of the most methodologically careful projects here, AnyJev and open-alternative-jev, train nothing at all. Both wrap a stock open model, read its option logits properly, and reach Jev's accuracy with better-calibrated probabilities. Both are single-benchmark, single-model-family results, so this settles nothing, but it does change what you should be asking of the nine trained open decision models.

→ Jump straight to the results

What is Jev?

This section covers only the details the comparisons depend on. TypeSafe's own docs and the four independent analyses in the appendix cover the rest.

There are three primitives (V):

Post-training is described as RLCD, reinforcement learning for calibrated decisions: an outcome-based objective against proper scoring rules rather than RLHF against human preference. The commercial envelope (V): model jev-1.13.0, $0.042 per million input tokens with output free, 64k context of which 32k is state plus the single longest question, text only, English first, and no fine-tuning or customisation of any kind. That last row matters more than it looks, because every open competitor's main lever is fine-tuning on your own labels.

confidence is a peakedness statistic, not a probability of being correct. For a choice over K options it is (K · p_max − 1) / (K − 1), normalised so a uniform distribution gives 0 and a one-hot gives 1. It says nothing about whether the argmax is right. TypeSafe say this themselves and return the full probabilities vector so you can compute your own, and two independent measurements suggest you should: anth.us finds AUROC 0.83 for ranking correct from incorrect but 7–15 points of overconfidence as a probability, and scienthoon finds the field "was never better than the max probability and sometimes much worse" (both T).

The vendor's own jaggedness page lists nine failure modes: literal reading of implied intent, arithmetic and counting, date comparison, indirection, large irrelevant state acting as a distractor, injected content, contradictory instructions, no guaranteed invariants between related questions, and generation. Their guidance — move arithmetic into code, filter the state, compare dates in code — amounts to saying Jev is a semantic verifier rather than a reasoner. I think that is a fair description and a reasonable thing to sell.

Why you would look for alternatives to Jev?

Cost is not the reason. At $0.042/Mtok input with output free, and JevBench's measured ~950 input tokens per decision (V + T), that is roughly 4 cents per 1,000 decisions. scienthoon's entire 4,621-call study cost about 6 cents. Check the arithmetic before you self-host to save money.

Four reasons that do hold up:

1. Trainability. Jev offers no fine-tuning, no LoRA and no per-account weights (V). You adapt through state and criteria only. Nine of the thirteen projects below take your labels, and nimble's jump from 66% to 90% on 2,676 examples shows how cheaply that can pay.

2. Latency. Hosted Jev is around 250 ms wall clock from a US or European origin, consistent across three independent measurements, against roughly 160 ms of upstream service time. A local encoder is 13–130 ms with no network involved.

3. Data residency. Zero data retention is an enterprise option. Running locally keeps the data on your own hardware.

4. Pinning and continuity. Aliases move, the docs say rate limits can change without notice (V), and the Vercel AI Gateway route cannot pin a version at all. That matters once you have tuned thresholds against one build.

The counterweight is that Jev is still ahead on knowledge-heavy decisions, which is where decider and kev both report their largest gaps, and that the alternatives were all days old when I wrote this.

How I picked what to include

A sweep of GitHub topic:system-one and topic:jev, a full-text repository search and a third-party index turned up roughly 40 projects beyond the obvious ones, plus about 20 "awesome-jev" link collections. Four criteria, all of which must hold:

1. It implements the System One contract: typed primitives with a probability distribution per question, read from the model rather than generated as text. A service that calls the hosted Jev API is an application, not a comparator.

2. It runs locally and reproducibly, with open weights or a stock open-weight base plus published inference code.

3. It is pinned and licensed.

4. It has been measured, or ships a bench that can measure it.

One warning about discovery aids: the widely-linked index at systemonemodels.org lists kev, decider and SemIf as having no published metrics, which is wrong for all three. Use it to find things, not to cite them.

Evidence grading

I grade every number and don't mix classes inside a comparison:

Almost every "X vs Jev" table in these repos compares a class S number against a class T or S number, measured on different items, prompts, dates and hardware. Several of the repos say so themselves. Pooling those numbers into one ranking would produce a ranking that is confidently wrong.

What was excluded, and why

The alternative field

The first table says what a project is; the second says whether you can use it. Row order is the same in both.

What each alternative is

Star counts are a poor guide as of 23 Sep 2026. laya gained 2,583 stars in the 24 hours between my two research passes, and the two most methodologically careful projects in the list have 68 and 51 stars. Sort by something else.

Whether you can use it

These columns decide whether you can adopt a project. They are currently scattered across thirteen READMEs, and collecting them is useful.

"Not published" means I went through the repo and the figure genuinely isn't there. Six of the thirteen don't state an option cap and five don't state a context limit, and those are exactly the numbers that decide whether you can adopt something. Most of these repos publish an accuracy number against Jev before they publish their limits.

Two things stand out:

• kev, von, decider and jeff implement TypeSafe's actual /v1/systemone wire format, so the official SDK points at them with an environment variable change. For adoption that matters more than any accuracy number. kev's CI runs TypeSafe's own example requests against its server.

• Check the option cap before you read the accuracy table. A 40-option routing problem rules out nimble, AnyJev and open-alternative-jev immediately, degrades laya, and is over jeff's default limit, whatever those projects score.

The projects

Each project follows the same template, and carries the repo link plus the exact commit and checkpoint revisions I read, because in a field this young an unpinned claim stops being checkable within days. Those pins are what I read, not what I recommend you run.

1. laya

github.com/NandhaKishorM/laya

Read 23 Sep 2026 at 725e020 · weights convaiinnovations/laya@1c5edc1, convaiinnovations/laya-typed-decisions@f9ab0b2

Three encoder checkpoints — ModernBERT-large 421M, mmBERT-base 322M, and a typed-decisions fine-tune — with a script-detecting router that picks between them in under 0.5 ms. 2.4 GB total, Apache-2.0.

laya documents its own limits more openly than most of the field does. The base checkpoints score near chance zero-shot on its own typed-decisions benchmark: 0.362 and 0.342, against a 0.318 random baseline and a 0.461 majority-class floor. The headline 0.766 comes from the checkpoint fine-tuned on that benchmark's own training split, and the authors say so (S).

Three other limits. Raw ECE is 0.466 and falls to 0.081 only after per-(type, option-count) temperature fitting, and laya-multilingual ships with no fitted temperatures. High-cardinality choice collapses — Banking77 0.425 against Jev's 0.870 — because ~77 labels share a 192–256 token head budget, about three to four tokens per label. Khmer scores 0.000 accuracy at 0.952 confidence, which is why the router exists.

Differs from Jev: encoder rather than decoder, multilingual, and designed to be specialised rather than used zero-shot. The authors call it "a fast base to specialise, not a zero-shot decision engine", which is the fairest self-description in this field.

Runs on: Mac (CPU/MPS) at 193–464 ms on CPU, 32.8 ms on a T4. It ships a Kaggle 2×T4 fine-tuning notebook.

2. laya-mlx

github.com/mizorewww/laya-mlx

Read 23 Sep 2026 at 0a85951 · weights aac6fef/laya-mlx@20aed81

An independent MLX port of all three laya checkpoints — encoder, decision transformer, scoring head and action head — with no PyTorch and no transformers runtime.

On an M3 Max in FP16, end to end including tokenisation: 13.42 ms P50 for one short question, 7.39 ms for the multilingual checkpoint, 146.8 and 395.0 questions per second at batch 50, under 1 GiB peak allocation (S, but unusually well documented).

Port fidelity is verified rather than asserted. All three checkpoints matched the upstream selected answer on 63/63 validation questions in both FP32 and FP16 — 378/378 comparisons — with pinned revisions, recorded weight hashes and 36 checksum-verified files. Almost no other port in this field documents fidelity at all.

Differs from Jev: it inherits laya's architecture and laya's weakness. It sets the latency floor for local decisions, but those decisions come from checkpoints that score near chance zero-shot, so latency is not the constraint that matters here.

Runs on: Apple Silicon only, macOS 14+.

3. kev

github.com/jaredpalmer/kev

Read 23 Sep 2026 at 557598f · weights jaredpalmer/kev-4b@485ace8, jaredpalmer/kev-0.8b@54f4f87

Qwen3.5-Base plus LoRA plus a pointer head that scores each option's </opt> hidden state against the question's <decide> hidden state. Three sizes: 0.8B, 4B and 9B.

Its harness, more than its model, makes it the most evaluation-ready project here: a genuine TypeSafe-compatible /v1/systemone, a benchmark runner that scores any System One endpoint over --remote, a live-Jev runner, and paired bootstrap confidence intervals between two saved runs. Its evals/external/ already holds two other projects' test sets in this format with their published live-Jev results.

It also reports a metric I did not see elsewhere: share of decisions automatable at a 5% error budget, 0.45–0.57 against Jev's 0.70 (S, but paired and on frozen items). Confident errors are 4.0% against Jev's 3.7%, after a fitted temperature of 2.1–2.4 baked into the checkpoint. Knowledge is where it loses, and the base model sets that gap: MMLU 0.74 against 0.90.

Weakness that shapes the local shortlist: there are no fast Metal kernels for Qwen3.5's Gated DeltaNet layers, so PyTorch runs reference code. Median on an M5, five questions, ~230-token state: 329 ms for 0.8B, 779 ms for 4B, about 2 s for 9B. The previous Qwen3 generation is three to six times faster on the same Mac (4B at 174 ms). An MLX backend is the next planned change.

Runs on: Mac, slowly, and any CUDA box.

4. SemIf (formerly OpenJev)

github.com/TheoLeeCJ/SemIf-OpenJev

Read 23 Sep 2026 at 1f2dea3 · weights Qwen/Qwen3.5-4B@851bf6e

Stock Qwen3.5-4B with no training at all, read through option logits. MLX, PyTorch/MPS and llama.cpp backends, all added in the last week.

It ranks around #2 to #4 on JevBench depending on the revision you read (T). On one RTX 3090, same model and same state, 21 binary criteria took 1.023 s through direct typed logits against 5.332 s for the autoregressive JSON equivalent, a 5.2× difference (S). The authors note that the two readouts agreed on only 18 of 21 criteria, so this is a systems comparison rather than a claim of semantic equivalence.

Differs from Jev: nothing was trained. That makes SemIf the control condition this field needs. It replaces "which fine-tune is best" with the more useful question of whether any of this training beats reading a stock model properly.

Runs on: Mac natively, three backends, ~8 GB in bf16.

5. GLiNER2

github.com/fastino-ai/GLiNER2

Read 23 Sep 2026 at 4abb613 · weights fastino/gliner2.5-base-v1@1a8bc24

The one pre-Jev entry (July 2025): a ~300M encoder for structured extraction and classification. It was pulled into this conversation because it was already producing schema-driven typed outputs from an encoder.

Unlike every other project here, GLiNER2 has no published System One-framed comparison against Jev. I am listing it because it belongs in the lineage, not because there is a number to cite.

Differs from Jev: extraction-first rather than decision-first, and no calibrated probability contract over a typed question set.

Runs on: anything, including CPU.

6. NanoJev

github.com/TianyuCodings/NanoJev

Read 23 Sep 2026 at 76fdfc9 · weights C-Tianyu/NanoJev@047b927

Qwen3-0.6B with decision heads, trained on 18,760 decision questions across four games: ViZDoom Basic, ViZDoom Predict Position, a 50×50 maze, and Snake.

It beats Jev decisively on those games — ViZDoom Basic 128/128 against Jev's 56/128, maze solved in 225 attempts against 2,738 (S) — because it is a domain-specialist embodied-control model, not a general text-decision model. Benchmarking it against text decisions would be as misleading as testing Jev on Snake. Its sibling project JevHarness, which generates LLM-authored task-specific Jev harnesses, is arguably more relevant to using Jev well.

Runs on: CUDA only, 21.6 GB of checkpoints, plus ViZDoom.

7. nimble (Bespoke Labs)

github.com/bespokelabsai/nimble

Read 23 Sep 2026 at 38edc3b · weights bespokelabs/Bespoke-Nimble-9B@e93fabc

Qwen3.5-9B plus LoRA r16, built in a day, trained on 2,676 contrastive examples. The curation method is worth stealing: write two near-identical examples that differ in one fact that flips the label, verify both with separate model calls, then verify that removing either evidence sentence makes the fact unknowable.

Holdout results on 324 items: Jev 93.21%, Nimble-9B 90.12%, Qwen3.8-27B 84.88%, base Qwen3.5-9B 66.36% (S). That jump from 66% to 90% on 2,676 examples is the strongest single argument in the field that a small local model closes most of the gap cheaply.

The authors state their constraints plainly: a 2,048-token prompt cap, ≤26 enum options through A–Z encoding, and probabilities that are explicitly uncalibrated at temperature 1.0. The labels are synthetic and, in their words, no person has reviewed them.

There is also a licence discrepancy. The model card says Apache-2.0, but the repository has no LICENSE file. Resolve that before you depend on it.

Runs on: Mac with 64 GB — the README says so explicitly — at 444 ms median on an M5 Pro, 106 ms on an H100. ~18 GB base plus a 0.17 GB LoRA merged on CPU.

8. von

github.com/wfzyx/von

Read 23 Sep 2026 at 14be898 · weights wfzyx/von@d8bb5e0

ModernBERT-Large 395M bidirectional encoder, dual cross-entropy and Brier loss, 250k NLI examples plus ~290k multi-domain, a fitted temperature of 1.1692, the TypeSafe wire format, Python and TypeScript SDKs, and a claimed ~18 ms GPU latency at a 3.2 GB download.

The README's headline and its own table disagree. The bullet list claims "SOTA Empirical Accuracy: 91.23% ... surpassing published commercial alternatives". The benchmark table two paragraphs below shows von at 72.0% macro against Jev's 96.6% on the same 49-task suite, a 24-point deficit. The ViZDoom row (von 9.00 kills against Jev's 5.62) is real, but it is one narrow zero-shot control task. The document does not support its own framing.

von is architecturally interesting, cheap to run and plausibly among the fastest local options, but every published claim needs independent verification before anyone cites it.

Runs on: Mac (MPS), CUDA, ROCm, CPU. Python ≥3.12.

9. decider

github.com/Mapika/decider

Read 23 Sep 2026 at 7557fe0 · weights Mapika/decider-2b@fa996ce

Two released sizes on Qwen3.5-Base, 1.9B and 34.7B-A3B, plus a 0.8B and a vision variant. 32k context, 2–255 choice options, 2–10 score levels: it matches Jev's declared envelope more closely than anything else here (V).

It reports third-party rankings rather than its own (T). On JevBench: Jev #1 at 75.4, decider-35b-a3b #10 at 68.9, decider-2b #21 at 64.6. On Decision Index v0.1, decider-35b-a3b is #4 of 32 and the highest-scoring trained model; the two entries above it are zero-training wrappers on stock 27B-class models.

It names where it loses: knowledge 0.51 against Jev's 0.69 per-area, and decider-2b's top-label ECE on JevBench hard items is 0.30, meaning it is confident where it is wrong. Its "Limits, stated plainly" section is the most candid in this list. One line from it: "rules written into the question are not followed at this size" — a one-sentence question scores 0.67, a paragraph of rules 0.24.

Apple Silicon is measured rather than asserted, and this is the only project in the list that did so. On an M1 Pro in float16, median request is 133 ms with the MPS patch against 171 ms without, and the MPS path scores 0.7553 accuracy / 0.0438 ECE against the published bf16 row's 0.756 / 0.041 on 1,500 held-out examples. That is a numerical-parity check on exactly this hardware class.

Runs on: pip install decider-ai, scripts/serve.sh gives /v1/systemone. 1.9B is 3.8 GB; 34.7B is 69 GB in bf16 and will not fit in 64 GB unified memory.

10. jeff

github.com/logan-markewich/jeff

Read 23 Sep 2026 at 34b32f9 · weights knowledgator/gliformer-large-v1@d0a4e53

A self-hosted drop-in for the Jev API on GLiFormer-large-v1 (400M). Device order is CUDA → MPS → CPU, so it works on a Mac unmodified, and the official SDK points at it with one environment variable. All three primitives.

It publishes where it loses. Measured on 1,600 labelled items across eight datasets: AG News 75.5% against Jev's 90.5%. JevBench v1.2.2 score 66.9, #9 of 18, against Jev's 75.3 — and the author notes that this rank comes from cost, because jeff is #14 of 18 on intelligence (T). Latency is 151 ms p50 on an L4 via Modal against Jev's 129 ms.

A small encoder that is six times cheaper (~$2.6 against ~$15.6 per million single-question requests) and fifteen accuracy points worse is a real point on the trade-off curve, and no other project in this list occupies it.

Runs on: Mac unmodified, ~0.8 GB.

11. AnyJev (Nokia Applied Research)

github.com/nokia-applied-research/AnyJev

Read 23 Sep 2026 at 54f1b35 · no weights of its own

AnyJev is a readout layer over any LLM rather than a model, and it trains nothing. Its thesis is that the problem is the readout, not the weights: max_tokens=1 plus logprobs already gives you a ranking, but one that moves when you reorder the options, and a confidence number you cannot threshold on. Both are properties of the readout, and both are fixable without a single label.

There are three levels, and every returned decision carries its level so downstream code can refuse to act on the wrong one. raw is a restricted softmax over label tokens. L0 adds cyclic-shift marginalisation — show K options in K rotations, combine in log space — plus prior correction, with no labels. L1 adds temperature scaling on 100–500 labels.

Qwen3-8B on BANKING77 20-way, n=300 (S):

All of that is on frozen weights, and an independent rerun came back bit-identical on every zero-label number.

The limitations section is exemplary, and it constrains how those numbers should be read: L0 is not a free win on every split, the batch prior needs at least 8 items of the same question, 26 options maximum, L1 does not survive distribution shift, coverage at a 5% risk budget is a high-variance point estimate at n=300, and everything so far is one model family.

Cost: a K-option choice costs K prefills (2 for noul, 1 for score), all sharing the state prefix and batchable. About 0.25 s per decision at batch 32 on an H100 with 20 permutations. Whether that multiplier is affordable on an M1 Max is an open question.

Runs on: whatever your base model runs on. No weights of its own.

12. open-alternative-jev

github.com/ikermoel/open-alternative-jev

Read 23 Sep 2026 at 4a85df1 · no weights of its own

Zero training. The state is written once and every question is answered from the next-token distribution at its own packed position.

The packing result is clean (S). On RACE-H, 250 passages × 4 questions: packed scoring gives 92.9% accuracy at 4.55 q/s using 186,898 tokens, against 92.6% at 1.66 q/s and 468,583 tokens for one question per forward pass. That is 2.5× fewer tokens and 2.3× the throughput at the same accuracy.

The headline result is on the 2,000-decision LocalLLaMA/typed-decisions suite: zero-shot Qwen3.6-27B at 73.7% accuracy and ECE 0.020, against Jev's 72.7% and 0.144. Both sit at the teacher's 73.5% self-agreement ceiling, so the accuracy column is saturated and calibration is the only place left where the models differ. The authors verified their scorer by reproducing laya's published numbers exactly before running anything else, which is what makes the rest credible.

The caution that goes with that number: JevBench ranked this project using the author's own option order (A. yes, B. no) and found the same model scored 21% instead of 72% with the options reversed. Read it alongside AnyJev, which exists to remove exactly that failure mode.

Runs on: your base model. ≤4B bases on a Mac; the 27B result was measured on an H200 MIG slice.

13. djev-dev

github.com/Davipar/djev-dev

Read 23 Sep 2026 at 3ce907e · weights google/diffusiongemma-26B-A4B-it@f7f5b7f

Documented, not runnable on a Mac. DiffusionGemma-26B-A4B through vLLM: it denoises all answer positions together rather than scoring a next-token distribution, with native image input and images as options. It adds no weights of its own.

It is the only architecturally different entry in this list, at around #5 on JevBench with 73.0 (T). A 26B MoE plus CUDA-only vLLM puts it out of reach here. If the comparison ever adds a "does the readout architecture matter" arm, this is the entry for it.

Where these converge, and where they split from Jev

Same as Jev. Typed primitives with probabilities read from the model rather than generated as text. One forward pass. Probabilities exposed so you can compute your own confidence. Four of thirteen match the wire format exactly.

Different from Jev, on four axes:

1. Trainability. Jev offers none. Nine of these can be fine-tuned on your own labels.

2. Calibration as shipped. kev bakes in T ≈ 2.1–2.4, von ships T = 1.1692, laya ships fitted temperatures for one checkpoint and none for another, nimble ships T = 1.0 untuned, and Jev ships whatever RLCD produced. Comparing raw ECE across these compares post-processing choices, not models. I think this is the single biggest reason the existing comparisons do not mean what they appear to mean.

3. Architecture. Encoders (laya, von, jeff, GLiNER2), decoder plus LoRA plus head (kev, nimble, decider, NanoJev), pure readout over a stock model (SemIf, AnyJev, open-alternative-jev), and diffusion (djev-dev). Jev is inferred to be a sparse MoE decoder with a linear head.

4. Scope. Jev is general text decisions. NanoJev is embodied control. GLiNER2 is extraction. djev-dev is multimodal.

The shared weakness: option order

Archer Hume measured a −0.28 mean log-odds shift in Jev from adding irrelevant options. JevBench recorded 72% against 21% from reversing two options. AnyJev measured a 0.230 raw flip rate. Nobody is immune, including Jev. Any comparison that runs each choice item in one option order is not measuring what it claims to measure.

Zero-training readouts are already competitive

AnyJev and open-alternative-jev, on different bases and different suites, converge on the same finding: a calibrated zero-training readout of a stock open model matches or beats Jev on the probabilities while matching or slightly trailing on accuracy. AnyJev moves auto-decidable traffic 6.8× on frozen weights. open-alternative-jev puts a zero-shot 27B at Jev's accuracy with seven times better ECE. AnyJev's re-measurement of laya on laya's own benchmark has the fine-tuned checkpoint winning on argmax (0.768) while its ECE is six times worse than a calibrated stock model's (0.215 against 0.036).

AnyJev also previews a closed-form head: a shrunk-LDA or ridge matrix fitted on the same hidden state from 200 labels per question, no gradients, seconds on a CPU, reaching 0.771 accuracy and 0.120 ECE against fine-tuned laya's 0.768 and 0.215.

Both results are single-model-family and single-benchmark, and both carry the order-sensitivity caveat above, so I would not treat this as settled. It does raise the question I want the second post to answer: if a properly-read stock model is already at Jev's level, what are the nine trained open decision models for?

Shortlist A — runs on an M1 Max with 64 GB

Criteria: open weights or a stock base, runs on MPS or MLX without patching, pinned and licensed, and has a published number I can check. If you want one answer rather than nine, start with decider-2b.

If you have labels, laya fine-tuned and nimble are the two built to be specialised, and both publish what specialisation bought them. On a smaller Mac, everything except nimble-9b fits in 16 GB.

One practical warning about dependencies: decider-ai pins numpy<2 and transformers>=5, laya wants transformers>=4.48, and laya-mlx wants neither. One virtualenv per model is mandatory. nimble's README already enforces three for the same reason.

These recommendations rest entirely on other people's numbers, and the second post is where I find out whether I would still give the same answer afterwards.

Shortlist B — needs a real GPU

None of these is required for the core comparison. One thing to know before you plan around free compute: two T4s do not pool into 32 GB, so Kaggle is good for ≤4B models, encoders and latency measurement on a known GPU, and is not a general escape hatch.

Key performance results

• open-alternative-jev — Slightly better accuracy than Jev: 73.7% vs 72.7%, with better-calibrated probabilities.

• AnyJev — Achieves similar accuracy to Jev without fine-tuning, and also produces well-calibrated confidence scores.

• djev-dev — Performs very close to Jev on JevBench: ~73 vs ~74–75.

• Nimble — A smaller model that still gets close to Jev: 90.1% vs 93.2% on its benchmark.

• SemIf-OpenJev — Uses no fine-tuning and still ranks around #2–#4 on JevBench.

Open questions, and what the second post will do

Assumptions in this post. Every number is published by someone else, and star counts and rankings were read on 22–23 September 2026. JevBench's Jev score already differs between my two reads, 75.4 and 74.4, which tells you how fast this is moving.

What still looks open, as far as I can find:

• I have not found a published run that puts the trained open models through one equal-calibration protocol on one frozen suite. If someone has done it since, that is the result I most want pointed at.

• I have not seen several of these run on one Mac with one request shape. The published Mac numbers I could find span four different Macs: M1 Pro, M3 Max, M5 and M5 Pro.

• "Does it know it doesn't know" is the least crowded question. Jev answers at ≥0.9 confidence on 9% of items whose deciding evidence was removed, kev-9b on 5%, kev-8b on 26%. scienthoon's unknowable-priority suite pins Jev at 44.7% accuracy while it states a mean probability of 0.74 on the level it chose.

• How much of any gap is the readout rather than the model.

The second post: one machine, one frozen suite, the same calibration budget for every model, at least two option orders per choice item, latency reported in families that are never combined into one ranking, and raw responses published.

Roughly forty projects appeared in a week, and any ranking published today is stale in two, which is why the second post is a protocol rather than a scoreboard.

If I have mis-read a project or graded a number wrongly, I would like to know. Several of these authors document their own weaknesses better than most commercial vendors do, and they deserve to be quoted accurately.

Appendix A — sources

Every number above traces to one of these. All public, all read on 22–23 September 2026.

TypeSafe (class V)

• typesafe.ai

• docs.typesafe.ai — quickstart

• docs.typesafe.ai — model jaggedness, jev-1.13

Independent analyses of Jev (class T)

• Archer Hume — Jev's architecture unmasked — black-box probing, tokenizer matching against 192 candidates, option-interaction experiments, MMLU ECE 0.0313

• Aman Arora — a first look at Jev — primitives, context split, 299 ms against 1,114 ms

• Aman Arora — an open-source Jev interface

• Arcturus Labs — Jev trades text generation for instant calibrated decisions — coin-flip calibration probe

• anth.us — can you trust Jev confidence? — AUROC 0.83, isotonic ECE 0.008

The projects

laya · laya-mlx · kev · SemIf · GLiNER2 · NanoJev · nimble · von · decider · jeff · AnyJev · open-alternative-jev · djev-dev

Benchmarks, harnesses and datasets

• JevBench — 534 frozen decisions, 66 ranked rows, hard tier hashed before any run

• Decision Index — 132,422 requests over 37 benchmarks, transformers engine with device=mps

• jev-ood-calibration — a regenerable 900-ticket synthetic suite with an unknowable-by-design label

• jev-benchmarks · decision-model-benchmark · JevHarness

• LocalLLaMA/typed-decisions — 400 cases × 5 typed questions, the de-facto community suite

• Luni/laya-jev-benchmark — third-party scorer with a Jev 1.13.0 row

Discovery aids

• systemonemodels.org/examples/alternatives — 20 listed reproductions, but it misreports kev, decider and SemIf as having no published metrics. Use it to find things, not to cite them.

• GitHub topic:system-one and topic:jev.

Unverified, and flagged as such

• Reports of "$5 free credit, no waitlist" for the TypeSafe console are secondary and SEO-flavoured; typesafe.ai still renders a waitlist element.

• nimble's repository has no LICENSE file although its model card says Apache-2.0. NanoJev's Hugging Face model-card licence is unset.

• JevBench's Jev score differs between my two reads (75.4 and 74.4). Re-read before citing.

Appendix B — live additions

List of useful Jev related links as discovered.

Summary

This is a broad technical/history lecture by Jeff Dean, described in the introduction as Google’s chief scientist and a co-lead of the Gemini project. His central argument is that modern AI did not emerge from one breakthrough. It is the cumulative result of improvements in scale, algorithms, architectures, hardware, distributed systems, training methods and inference techniques, with these improvements multiplying rather than merely adding together.

The talk moves from Dean's early neural-network work through distributed training, word embeddings, seq2seq, TPUs, Transformers, sparse/MoE models, Pathways, inference-time reasoning, distillation, reinforcement learning and speculative decoding, before explaining how these pieces come together in Gemini and where he thinks AI research is heading.

The talk in one sentence: Modern AI is best understood as a co-evolving stack—models, algorithms, data, distributed systems and specialised hardware—and the next phase shifts much of the emphasis from merely training bigger models toward efficient inference, long-term retrieval/memory, multimodal world models, reasoning, and coordinated populations of AI agents.

Key points

1. Scale remains important, but scale alone is the wrong explanation for recent AI progress. Dean argues that increasing compute/data/model size has produced fairly continuous gains for roughly 13–14 years, but architectural and algorithmic improvements have been equally important. His example is that a 20× scaling improvement combined with a 50× algorithmic improvement can yield something closer to a 1,000× overall improvement.

2. Distributed training was foundational very early. Dean's undergraduate work already explored what would now be called model and data parallelism. At Google, this developed into systems capable of training neural networks 50–100× larger than contemporary models using many asynchronous model replicas and distributed parameter servers.

3. Representation learning showed that models could discover meaningful concepts without explicit labels. Google's large unsupervised vision experiments on 10 million YouTube frames produced high-level units responsive to things such as cats, faces and people. Using the learned representation to initialise supervised training produced a large improvement on ImageNet-22K. Word-vector work similarly demonstrated that semantic relationships naturally emerge geometrically in embedding space.

4. Specialised AI hardware became necessary because inference itself was becoming economically impossible on conventional compute. Dean describes estimating that deploying a much better speech model to one billion users for three minutes per day would require roughly doubling Google's entire computer fleet. That helped motivate TPUs. TPUv1 exploited reduced precision and the fact that neural networks are dominated by a relatively small set of linear-algebra operations.

   In the figures given in the talk, TPUv1 was 15–30× faster and 30–80× more energy efficient than contemporary CPUs/GPUs. Later TPU systems evolved into interconnected ML supercomputers; Dean says Ironwood delivers roughly 3,600× the pod-level performance of TPUv2 and ~30× the FLOPs/watt.

5. Transformers solved two major limitations of recurrent language models. LSTMs were inherently sequential and compressed everything encountered so far into one state vector. Transformers instead retain representations and use learnable attention to access them, making computation much more parallelisable and information easier to retrieve. Dean presents the Transformer as dramatically more compute-efficient than comparable LSTMs.

6. Self-supervised learning unlocked essentially unlimited training data. Predicting the next token or masked tokens gives a precise training signal without requiring humans to label data. Autoregressive prediction naturally became the basis for generative/chat models, while fill-in-the-blank objectives remain particularly useful for representation learning.

7. Sparse / mixture-of-experts models are a major part of the scaling story. Rather than activating every parameter for every token, sparse models learn specialised components and a routing mechanism that selects which experts to activate. This allows much greater model capacity without proportional inference cost. Dean cites an example giving around an 8× reduction in training compute for equivalent accuracy.

8. At extreme scale, reliability becomes an ML problem as well as a systems problem. Google's Pathways attempts to present thousands of accelerators as one giant computer. At this scale, silent data corruption becomes significant: hardware can occasionally return incorrect values rather than simply fail. Google monitors training signals such as gradient norms, uses deterministic replay to distinguish anomalous data from hardware errors, and swaps faulty hardware for hot spares.

9. Inference-time compute is now another scaling axis. Dean frames chain-of-thought-style reasoning as giving the model additional computation at inference time: each additional generated token represents another model pass. This substantially improved mathematical problem solving once models became sufficiently capable.

   The later Q&A extends this idea: for critical tasks, generate multiple candidate solutions, have the model evaluate them, and retain the strongest result. Dean says this can reduce hallucination rates, but it makes inference efficiency even more important.

10. Distillation is central to making powerful models economical. Instead of training a smaller model only against hard labels, the teacher's full probability distribution supplies much richer supervision. Dean explicitly says this is used in Gemini to move capability from Pro-scale models into smaller Flash-scale models.

11. RL/post-training increasingly determines what capabilities a pretrained model actually exhibits. Reward signals can come from humans, another model, or objectively verifiable outcomes such as compiling code, passing unit tests or proving a theorem. Dean identifies RL for non-verifiable domains as an important unresolved research problem: where do sufficiently reliable reward signals come from?

12. Speculative decoding addresses the sequential bottleneck of autoregressive generation. A cheap draft model proposes several tokens, while the large target model checks them in parallel. Importantly, Dean stresses that this requires no architecture change or retraining and preserves the target model's output distribution while improving utilisation.

Gemini: the synthesis of these ideas

Dean describes Gemini as an attempt beginning in February 2023 to consolidate previously separate Google language and multimodal efforts into one major project. The design goal was multimodality from the beginning: text, images, audio and video, with newer systems also exposed to things such as LiDAR and robotic-control data.

Gemini therefore isn't presented as one novel architecture. It is a system incorporating essentially the whole preceding history:

TPUs + distributed/model/data parallelism + cross-datacenter training + Pathways + JAX + Transformers + sparse models + distillation + long context + inference-time reasoning + speculative decoding + SFT + RL.

One particularly interesting point is Dean's view of context vs model weights. Information absorbed into parameters from trillions of training tokens becomes somewhat "muddled and fuzzy"; information supplied directly in context remains precise. That is part of Google's motivation for pushing very long context windows.

Google's desired model progression is also notable:

Next-generation Flash ≳ previous-generation Pro

The idea is that frontier-level capability should repeatedly migrate into a much cheaper model tier, making yesterday's expensive capabilities economical for mainstream applications.

Capabilities he highlights

The talk uses several examples to illustrate how rapidly the nature of model capability is changing:

• Mathematical reasoning: a general-purpose Gemini Pro-scale model, given a large inference-time thinking budget, achieved a gold-medal-level result on the IMO rather than relying on the previous collection of specialised theorem-proving/geometry systems.

• Generative UI/software: a model can take source material and dynamically construct an interactive visual explanation or complete website. Dean expects this to mean far more software gets created, including by people unable to program conventionally.

• Visual reasoning: image-generation models can reason in "pixel space", generating intermediate visual states of a physical problem rather than reasoning solely through language.

• World models: he treats world modelling as closely related to multimodal understanding. Gemini-derived systems can generate persistent interactive worlds, which can in turn produce unusual simulated scenarios for systems such as autonomous vehicles.

Most important forward-looking ideas

1. Humans managing teams of AI agents

Dean expects the dominant interaction model to move beyond:

1 person → 1 chatbot

towards something more like:

1 person → dozens or hundreds of AI agents

That raises new problems around HCI, delegation, coordination, communication between agents and converting relatively weakly specified human goals into reliable execution.

This is probably the strongest forward-looking theme in the talk.

2. Context windows alone won't solve long-term memory

A million tokens is useful, but Dean asks what happens when the useful information is effectively a trillion tokens.

His likely architecture is hybrid:

huge corpus → learned retrieval → lightweight relevance filtering → small highly relevant subset → context window → model

Potential applications include a Gemini able—with permission—to reason over all of someone's email/photos, multimodal search across huge video collections, or coding agents able to draw upon an entire corporate codebase.

This is effectively a prediction that retrieval and context management become core parts of the model system, rather than endlessly increasing raw context length.

3. Inference efficiency may matter more than training efficiency

As agents proliferate and invoke other agents, inference volume compounds. Dean therefore expects:

• specialised inference hardware,

• better inference algorithms,

• aggressive latency optimisation,

• AI-assisted chip design.

He specifically contrasts an interaction responding in roughly 100 ms versus 5 seconds: latency becomes a product capability, not merely an infrastructure metric.

4. AI research does not necessarily require frontier-scale compute

His final answer is particularly relevant for researchers without thousands of accelerators. He recommends:

• test genuinely different ideas at very small scale;

• run several scales;

• study the slope/scaling trend, rather than obsessing over absolute benchmark position;

• favour novel approaches with promising scaling behaviour over tiny improvements to the current SOTA.

An idea slightly below the baseline at tiny scale but improving faster may be much more important than an idea that narrowly beats the baseline at one small scale.

That is essentially an argument for researching scaling behaviour rather than leaderboard points.

His overall view of AI risk and impact

Dean is relatively optimistic about catastrophic AI safety concerns, saying he believes careful engineering can constrain what systems are permitted to do. His nearer-term concerns are more concrete:

• highly convincing AI-generated misinformation/audio/video;

• managing labour and skill transitions;

• ensuring people learn to use AI tools rather than simply being displaced by automation.

He expects major effects across employment, education, healthcare, misinformation/media, governance/national security, entertainment and AI-for-science, and sees widespread access to previously scarce expertise as one of the major benefits.

Appendix: Further reading

Overview

• A Golden Decade of Deep Learning: Computing Systems & Applications (Jeff Dean, Daedalus, 2022) — Dean's own written account of much of the same history: scale, hardware, and the applications they unlocked.

Early foundations: distributed training and representation learning

• Large Scale Distributed Deep Networks (Dean et al., NeurIPS 2012) — the DistBelief paper: asynchronous model replicas and parameter servers for training much larger networks.

• Building High-level Features Using Large Scale Unsupervised Learning (Le et al., 2012) — the YouTube-frames "cat neuron" experiment and its ImageNet-22K result.

• Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013) — word2vec, where semantic relationships emerge geometrically in embedding space.

• Sequence to Sequence Learning with Neural Networks (Sutskever, Vinyals & Le, 2014) — the LSTM encoder-decoder approach that Transformers later superseded.

Hardware and systems

• In-Datacenter Performance Analysis of a Tensor Processing Unit (Jouppi et al., 2017) — the TPUv1 paper, source of the speed and energy-efficiency comparisons against CPUs/GPUs.

• Ironwood: The first Google TPU for the age of inference (Google, 2025) — the latest TPU generation Dean cites, and its inference-first framing.

• Pathways: Asynchronous Distributed Dataflow for ML (Barham et al., 2022) — the system for presenting thousands of accelerators as one computer.

Architectures and efficiency techniques

• Attention Is All You Need (Vaswani et al., 2017) — the original Transformer paper, replacing recurrence with attention.

• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al., 2017) — the sparse MoE layer with learned routing that underpins the sparse-model scaling story.

• Distilling the Knowledge in a Neural Network (Hinton, Vinyals & Dean, 2015) — the distillation technique Dean describes for moving Pro-scale capability into Flash-scale models.

• Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman & Matias, 2022) — draft-and-verify decoding that speeds up generation without changing the output distribution.

Reasoning and Gemini

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022) — the paper behind treating intermediate reasoning tokens as extra inference-time compute.

• Gemini: A Family of Highly Capable Multimodal Models (Gemini Team, 2023) — the technical report for the natively multimodal model that pulls these ideas together.

• Gemini Deep Think achieves gold-medal standard at the IMO (Google DeepMind, 2025) — the IMO result Dean highlights, reached by a general model with a large thinking budget.

1. Two common assumptions about embedding size

Many of us bring two assumptions to choosing an embedding size.

The first assumption: more dimensions means a richer vector. Twice the coordinates should mean twice the room for visual detail, so 3072 dimensions must hold more of a photograph than 768 does.

The second is the cross-model version: Gemini Embedding 2 outputs 3072 dimensions and some other model outputs 768, so Gemini has more capacity to work with.

The first assumption has a measurable cost. Take the example I will return to throughout this post: a real-estate search system over 10 million property photos. At float32, 3072-d vectors take 122.88 GB of raw payload before any index overhead. The same catalogue at 768-d takes 30.72 GB. That is roughly a 92 GB difference from a single architectural choice you probably made in an afternoon.

If the assumption holds, that extra storage buys better search results. If it does not, the storage buys nothing.

2. The dimension/quality questions

When the same model is served at 3072, 1024 and 256 dimensions, what happens to text→image retrieval quality?

And does the published evidence cover the models you are most likely to deploy?

The first question has a good answer, from two independent labs. The second does not: for Gemini Embedding 2 and Cohere Embed 4 — likely two models on your shortlist — nobody has published the per-dimension curve for image retrieval. I could not find one at all. That gap is half of what this post is about, so I want to state it up front rather than save it for the end.

3. The one commercial ablation that answers the question: Amazon Nova MME

Amazon Nova Multimodal Embeddings is the only major commercial multimodal model I found with a published per-dimension table for text→image retrieval. The architecture, training and benchmarks are held constant; only the vector length changes. Amazon states that the shorter vectors are prefixes of the full 3072-d representation, trained so the front of the vector carries the most signal. Section 5 explains why that works.

Three results stand out.

3072 → 1024 is cheap. Removing two-thirds of the coordinates costs 1.0 points on TextCaps and 1.1 on MSCOCO.

1024 → 384 costs noticeably more. Against the 3072 baseline: 3.3 points on TextCaps, 3.8 on MSCOCO, 5.3 on ViDoRe.

256 loses substantial quality. It costs 5.8, 6.1 and 8.5 points respectively, or roughly 6.5%, 8.0% and 14.5% in relative terms.

The three benchmarks do not degrade at the same rate. ViDoRe (Visual Document Retrieval, a benchmark of document screenshots with dense text and fine visual detail) loses 8.5 points at 256 dimensions where TextCaps loses 5.8. This is the first sign that visually rich retrieval needs more dimensions than object-centric captioning benchmarks reveal. I think that matters for property photos, and section 7 returns to it.

One note on metrics if you plan to reproduce any of this: the image-retrieval columns are average Recall@1/5/10 (R@K is the share of queries where the correct image appears in the top K), and ViDoRe is NDCG@5, a graded ranking score over the top five results. Do not compare these figures against benchmarks that quote Recall@1 alone.

4. Open-weight corroboration: Jina CLIP v2

One vendor's ablation of its own model is not a pattern. Jina CLIP v2 serves as a control: the weights are open, text→image Recall@5 is published at six lengths, and you can rerun the evaluation yourself.

768 dimensions is not worse than 1024 here, and on Crossmodal-3600 it is nearly a point better. Even 256 stays within about one point of full width on all three tests. Quality drops sharply at 128 (3-4 points below full width) and 64 (7-9 points).

This is the same curve shape as Nova's, at a different scale, from a different lab, on different benchmarks. Two independent sources agreeing is the main argument of this post.

5. Why the curve has this shape

Both models are trained with Matryoshka Representation Learning (MRL). The training objective is applied not only to the full vector but to nested prefixes of it, so the first 256 coordinates are optimised to work as an embedding on their own, the first 768 likewise, and so on. This means that the dimensions of an MRL-trained model are not equally informative independent coordinates. Truncating is not compressing a flat 3072-d space; it is taking a prefix of a representation designed to survive truncation.

This also explains why the cross-model assumption in section 1 fails. Gemini Embedding 2 at 768 dimensions is still the full Gemini network doing the work, which is not the same as swapping in a smaller 768-d vision-language architecture. Dimension count is not comparable across model families, only within one.

One reproducibility trap: taking a prefix changes the vector's norm. If you compare an unnormalised truncation against a normalised full vector, you will manufacture a dimensionality effect that is not there. Gemini's API re-normalises automatically when you request a non-default size. If you truncate by hand, L2-normalise before any cosine comparison.

6. The gap: the two models you may be evaluating

The missing evidence below is, I think, the most useful thing this post has to say.

Gemini Embedding 2. What is published is genuinely good: the model is natively multimodal rather than a text model bolted onto a vision encoder, MRL losses are trained at 768- and 1536-d prefixes, and absolute text→image Recall@1 is strong (mean 80.5; DOCCI 93.4, TextCaps 89.6, MSCOCO 62.9). DOCCI (Descriptions of Connected and Contrasting Images) and TextCaps are detailed-caption datasets rather than short object labels. I expect detailed-caption retrieval to map better onto a query like "sunlit open-plan kitchen with a waterfall-edge stone island and black pendant lights" than COCO's dominant-object framing does, but that is my inference about the application rather than something Google measured.

Google has not published any 3072 vs 1536 vs 768 text→image table. The dimension ablation in Google's docs covers Gemini Embedding 001, a text model, evaluated on MTEB (1536: 68.17, 768: 67.99, 256: 66.19, 128: 63.31). That is encouraging for MRL in general and consistent with everything above, but it is not evidence about Embedding 2 on complex images, and I would rather say so than treat it as a substitute.

Cohere Embed 4. Embed 4 is genuinely multimodal, with clean MRL sizes. Its distinctive capability is fusing image and text into a single vector, which is attractive for listings where room type, suburb or other structured attributes are reliable text that you would otherwise force the pixels to carry. There is again no published per-dimension image-retrieval curve.

One external datapoint exists, and it needs careful handling. Amazon's Nova report benchmarked Embed 4 through the public Bedrock API and scored it 22.9 on MSCOCO against Nova's 76.7. Three caveats come with that number: Amazon is a direct competitor, the figures are not Cohere self-reported, and the Embed 4 output dimension used is not stated. I read it as a reason to benchmark Cohere yourself, not as evidence about Cohere's dimension curve, which it says nothing about.

7. Input resolution probably matters more than dimension

I have spent a thousand words on dimension count. The published evidence points to input resolution as the larger effect.

Jina CLIP v2 ablates image resolution separately, and the effect is much larger than anything in sections 3 or 4. On ViDoRe, moving from 224 to 384 pixels lifts average NDCG@5 from 0.256 to 0.454. Raising resolution to 512 pixels improves it further; going from 512 to 768 costs 2.25x the image patches for a gain of 0.019, which is why the authors settle on 512.

Compare the two effects directly: cutting a well-trained MRL vector from 1024 to 512 dimensions costs a fraction of a point, while under-resolving the image costs a large share of retrieval quality. Preprocessing can destroy the information before vector width ever becomes the binding constraint.

This matters most for queries that hinge on a small local feature:

"gas cooktop beneath a concealed rangehood" · "herringbone timber flooring" · "frameless shower with niche"

Check what your API does here: Cohere documents that Embed v4 downsamples images above 2,458,624 pixels (1568×1568 square), while Gemini's embedding docs publish no equivalent threshold. One caveat: ViDoRe is document screenshots, not property photography, so do not transfer the specific 512px optimum. What transfers is the ordering of the two effects: fix resolution first, then choose a dimension.

One failure mode is unaffected by either setting: an embedding can correctly identify an image as a kitchen and still rank it top for "white kitchen, black island" when it is a black kitchen with a white island. Attribute binding is what COCO-style benchmarks underexpose.

8. The corrected mental model

Pulling the findings together:

1. Choose the model family first, then the dimension. Reversing the order risks picking a weaker model because it exposes more coordinates.

2. Within an MRL model, 3-4x compression plausibly costs about a point. Nova and Jina agree on this from different labs and at different scales.

3. The cliff is real, and it sits well below the headline width. Nova breaks at 256, Jina at 128.

4. Fine-grained visual retrieval degrades faster than object-centric retrieval. ViDoRe falls fastest in Nova's table, and I would treat property photography as nearer that end.

5. Resolution before dimension. It is the larger measured effect in the published evidence.

6. Dimension and numeric precision are separate axes. Cohere exposes int8 and binary output; those compound with width rather than substituting for it, and conflating the two will confuse your results. It is a real axis, but it needs its own post.

My current view on the operating regions — a hypothesis to test on your own corpus, not a set of guaranteed scores: 3072→1536 is rarely justifiable on quality alone; 1536/1024→768/512 is usually the attractive region; 384/256 is viable when memory-constrained but increasingly lossy on complex imagery; 128/64 is too aggressive without a reranker behind it.

That returns us to the 92 GB savings:

Those figures are arithmetic on raw float32 payload only. A real ANN index also carries graph or product-quantisation structures, IDs and metadata, and latency has to be measured in your target vector database: dimension is a first-order effect on distance computation and memory traffic, but traversal, cache behaviour and I/O can dominate.

The conclusion the table supports: Gemini at 768 instead of 3072 dimensions saves roughly 92 GB of raw vectors on a 10M-photo catalogue, at one of Google's own recommended MRL sizes. That is why a sub-point quality loss deserves this much attention.

9. Limits of this analysis

Four limits I want on the record.

• Every number here comes from a general benchmark. None is a property-image benchmark.

• Nova and Jina agreeing is suggestive, but it does not guarantee that Gemini's or Cohere's curves have the same shape. They may well not.

• The 768 starting point for Gemini is extrapolated from Google's MRL training sizes plus Nova and Jina's behaviour. I could not find it demonstrated anywhere.

• Cliff location is model-specific. You will have to find your own, and a few hundred well-judged domain queries will tell you more than another general benchmark will.

That reverses the question we started with. The useful question was never how many dimensions a model has. It is how far down that model's own curve you can go before your queries degrade.

Appendix A: Primary sources

These are where the numbers in this post come from. I have noted which sections lean on each one so you can check the claims against the original.

1. Amazon Nova Multimodal Embeddings technical report. Amazon. cdn.amazon.science

   Per-dimension text→image ablation at 3072 / 1024 / 384 / 256 (section 3), the prefix-truncation design (section 3), and the Cohere Embed 4 comparison run through Bedrock (section 6).

2. jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images. Jina AI. arXiv:2412.08802

   Recall@5 across six MRL lengths (section 4) and the image-resolution ablation on ViDoRe (section 7).

3. Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini. Google. arXiv:2605.27295

   Native multimodal architecture, MRL training at 768- and 1536-d prefixes, and absolute text→image Recall@1 on DOCCI, TextCaps and MSCOCO (section 6).

4. Embeddings, Gemini API documentation. Google AI for Developers. ai.google.dev

   Supported range and recommended sizes (section 6), automatic re-normalisation at non-default sizes (section 5), and the Gemini Embedding 001 MTEB dimension table (section 6).

5. Introduction to Embeddings at Cohere. Cohere documentation. docs.cohere.com

   Embed 4 dimension options (section 6), the image downsampling threshold (section 7), and int8 / binary output types (section 8).

6. Embed: Secure AI Retrieval. Cohere product page. cohere.com/embed

   Embed 4's multimodal positioning and fused image-and-text vectors (section 6).

Appendix B: Background technique and benchmarks

Useful if you want the original definition of a method or benchmark mentioned above. None of the post's figures come directly from these.

• Matryoshka Representation Learning. Kusupati et al., 2022. arXiv:2205.13147. The training method behind truncatable embeddings (section 5).

• ColPali: Efficient Document Retrieval with Vision Language Models. Faysse et al., 2024. arXiv:2407.01449. Introduces the ViDoRe benchmark (sections 3 and 7).

• TextCaps: a Dataset for Image Captioning with Reading Comprehension. Sidorov et al., 2020. arXiv:2003.12462 (sections 3 and 6).

• Microsoft COCO: Common Objects in Context. Lin et al., 2014. arXiv:1405.0312 (sections 3 and 6).

• DOCCI: Descriptions of Connected and Contrasting Images. Onoe et al., 2024. arXiv:2404.19753 (section 6).

• Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset. Thapliyal et al., 2022. arXiv:2205.12522 (section 4).

• Towards Zero-shot Cross-lingual Image Retrieval. Aggarwal and Kale, 2020. arXiv:2012.05107. Introduces XTD10 (section 4).

• MTEB: Massive Text Embedding Benchmark. Muennighoff et al., 2022. arXiv:2210.07316 (section 6).