MindSage engineering notes · 03
Testing embedding models on a hand-labelled journal
Every speed benchmark I had would stay green while search quietly returned the wrong entries. So I built a small labelled test set, and found that my default model ranked the wrong entry first more than half the time.
MindSage finds journal entries by meaning, not keywords. Each entry is turned into a vector by an embedding model, and a question is matched against those vectors. The chat feature (post 02) feeds its top five results to a language model, so if search is wrong, the answer is confidently wrong too.
Speed benchmarks can't see any of that. You can swap a model, change a threshold or break the indexing, and every latency number looks fine while the feature stops working. I needed a test that measures whether the right entry comes back.
The test set
I wrote 18 short journal entries, each on a clearly different subject: a running injury, a call with a sister, a failed deployment at work, insomnia, a mum's new pottery class, a dal recipe, a dentist visit, and so on. Then I wrote 15 questions and labelled by hand which entries answer each one.
The subjects are deliberately distinct. With near-duplicate entries a low score would say nothing about the code. With distinct ones, a working system should get the right entry first almost every time. This is not a research benchmark. It's a regression guard that answers "did this change break search?"
The labels are committed to the repo, so the judgement is made once and every run is scored against the same answers.
Three numbers
- precision@1: how often the top result is a right answer. This is what the user sees first.
- recall@5: how much of the right material is in the top five. This is what the chat model gets to read.
- MRR (mean reciprocal rank): rewards getting the right entry near the top, not just somewhere in the list.
The baseline was bad
With nomic-embed-text:v1.5, the model the app ships with, recall@5 was 0.767, MRR 0.644 and precision@1 0.467. The right entry was usually somewhere in the top five, but the top hit was wrong in 8 of 15 questions. These weren't judgement calls:
| Question | Top result | Should have been |
|---|---|---|
| "What have I written about my knee?" | the dentist entry | the running injury |
| "What is my mum up to these days?" | a friend moving away | the pottery class |
| "Anything about cooking or food I made" | the pottery class | the dal recipe |
| "Have I been to any medical appointments?" | a river walk | the dentist |
Hypothesis 1: missing task prefixes. Wrong.
The nomic model's documentation says to prefix stored text with search_document: and questions with search_query:. The app didn't do either. That looked like the obvious bug.
Adding the prefixes made every number worse: precision@1 dropped from 0.467 to 0.333 and MRR from 0.644 to 0.544. My best guess is that the way the model is packaged already handles this, but I didn't dig further. What matters is that the "obvious fix" from the docs would have shipped a regression without this test. It's written down in the repo so nobody spends an afternoon rediscovering it.
Hypothesis 2: a different model
Next I ran Google's embeddinggemma on the same entries, questions and labels. Only the model changed.
Wrong top hits went from 8 of 15 to 4 of 15, and the clearly wrong ones were among those fixed. "My knee" now returns the running injury, and "my mum" returns the pottery class.
It's also nearly a drop-in. Both models produce 768-number vectors, so the vector table doesn't change. Every existing entry would still need re-embedding, because vectors from two models can't be compared.
What it costs
| nomic | embeddinggemma | |
|---|---|---|
| Model on disk | 274 MB | 621 MB |
| Embeddings per second | 9.44 | 7.53 |
| Re-index 5,000 entries | 8.8 min | 11.1 min |
| precision@1 | 0.467 | 0.733 |
So the trade is 347 MB of the user's disk and about 2 extra minutes of background indexing at 5,000 entries, for a top result that's right 73% of the time instead of 47%. I think that's a good trade. It's a product decision, though, not a benchmark result, and I haven't shipped it yet.
How much to trust this
Not as much as the percentages suggest. precision@1 going from 0.467 to 0.733 means 7 correct top hits became 11. That's four questions. The direction is consistent across all three metrics, and the misses that disappeared were unambiguous, so I trust the conclusion. I wouldn't quote "+57%" as if it came from a large evaluation.
One thing I do trust fully: the test is deterministic. Separate runs on different days reproduced the baseline exactly, 0.767 / 0.644 / 0.467 to three decimals. That's what makes it useful as a guard. If those numbers move, something changed.
Next
- Embed a title plus summary instead of the whole entry. Long entries dilute the vector, and questions are short.
- Hybrid search. A plain keyword match would get "my knee" right instantly. SQLite's FTS5 can sit next to the vector table in the same file.
- Grow the test set with harder, overlapping entries, so it can tell good from very good.
Takeaways
- If a feature depends on search, test search quality, not just speed. Speed tests stay green while quality breaks.
- Eighteen hand-written entries were enough to catch a bad default, a wrong fix from the docs, and a better model.
- Say how small your sample is. It makes the conclusion more believable, not less.