MindSage engineering notes · 05

A benchmark harness that's hard to fool, starting with me

Before optimising my Electron app, I built a benchmark suite with eleven stages. The best things it did were refusing to let me type numbers into docs, showing regressions first, and catching three of my own wrong conclusions.

By Arinay · November 2026 · About 8 minutes

The other posts in this series quote a lot of numbers: list queries dropping from 246 ms to 0.28 ms, chat taking 14 seconds, search quality of 0.467. Every one of them came from the same harness. This post is about how it's built, and mostly about the ways it stopped me from fooling myself.

What it measures

npm run bench runs up to eleven stages against MindSage, a Windows journaling app built with Electron, SQLite and local AI models:

• Database: every query the screens make, at 150, 5,000 and 50,000 entries, plus reads while a background thread writes.
• AI: entry enrichment, chat, inline suggestions, cold model load, embedding speed.
• Chat pipeline: each of the four stages separately.
• Retrieval quality: a hand-labelled test set, so search can't get faster and worse at once.
• Vector search, speech-to-text, media over IPC.
• The packaged app: cold start, IPC round trips, dashboard render time, dropped frames while scrolling, memory across repeated navigation.
• Size: installer, packaged output, and JavaScript bundle bytes attributed to each npm package by decoding source maps.

A full run takes about 40 minutes. A database-only run takes about 6.

Rule 1: percentiles, not averages

The stats helper's opening comment explains this better than I can:

A mean hides the stalls that actually get noticed. One 400 ms freeze inside fifty 8 ms calls averages out to 16 ms and looks fine, while p95 shows it.

Every result reports p50 and p95. Warm-up runs are thrown away, because the first call to a prepared statement pays for query planning once, and that's not what a user feels while scrolling. Each scenario has a 20-second budget, so a 3-second query at 50,000 entries can't stall the suite. That budget has a cost too: one early number rested on only 13 samples, and the docs say so next to it.

Rule 2: nobody types a number

Every run is saved as JSON under a label like after-wal-and-indexes. The reports, the comparisons and the status board are all generated from those files. The issue list doesn't store numbers either. It stores where each issue's metric lives in a run:

{
  "id": "DB-1",
  "title": "No indexes on `journal_entries`",
  "severity": "high",
  "metricLabel": "`list.page1` p95 @ 50k",
  "metric": {
    "stage": "db",
    "volume": 50000,
    "path": ["results", "list.page1", "p95"],
    "unit": "ms",
    "target": 20
  }
}

So the board can't go stale and can't be "rounded up" by hand. The human part of each entry is the reasoning: what changed, why, and what it means.

Rule 3: regressions come first

The comparison report doesn't open with the wins. If anything got slower, the first thing on the page is a warning box listing it, ending with one line:

A win elsewhere does not cancel these. Decide deliberately whether each is an acceptable trade, and record the decision in the optimisation log.

That rule made me write down an accepted regression when an index made the main list 800× faster and a gallery query 30% slower. Without the rule, I'd have only seen the 800×.

Rule 4: know your noise

Early on, a change looked like it made writes 2.1× slower and background contention 2× slower, which fit a neat theory about foreign keys. The same comparison showed contention 2.5× faster at another size, which fit nothing.

So I compared the same branch with itself, no code change at all. The report said writes were 13.2× slower and listed twelve regressions. On my laptop, a single scenario can move about 2× between identical runs. Every delta in the real comparison was inside that, so it proved nothing in either direction. The log records "no detectable effect" rather than a result.

Two more ways the harness protects comparisons:

• Every report prints the machine. When a Windows update landed mid-project, I re-measured unchanged code across it. The large database scenarios came out flat, so the update wasn't a confound.
• Every run records the AI models. Swapping a model moves AI timings more than any code change, so an AI comparison only counts with the models held fixed.

Rule 5: a quick check on every commit, that never blocks

The pre-commit hook runs a 25-second benchmark: the database stage at 150 and 5,000 entries, 20 samples each, compared with the previous quick run. It only runs when the commit touches code the app executes, and skips merges and rebases.

It never fails a commit. With 2× noise and 20 samples, a hard gate would refuse good commits. It prints regressions loudly instead.

It earned its place. When I added an index to the journal table, the quick run showed the list query going from 9 ms to 98 ms. The index alone made things worse because of how the query grouped rows. The full story is in post 01. Without the hook I would have found that much later.

Three times it proved me wrong

1. "The database will throw busy errors." My own performance audit predicted it. The harness recorded zero. The real problem was readers silently waiting 200 ms behind a writer.
2. "Sending images over IPC is a major problem." The audit rated it as major. Measured, a full image crossed IPC in about 2 ms. I closed it with no work.
3. "The renderer is 76% of startup." That came from one early run. Three clean runs showed the opposite: the search server I bundled at the time took about two thirds of startup. The single-run number has a correction note next to it in the findings.

And once the harness itself was wrong. After I removed an old login argument from the app, the benchmark kept passing it. Every app-level scenario was timing an {"error": "Invalid token"} response. The dashboard read "0.30 ms, 25 bytes", which should have been the giveaway. Now I check payload sizes next to timings: if a response is suspiciously small, it's probably an error.

Writing down what's fine

The optimisation log has a section called "Measured fine, do not optimise": chat time to first token, cold start, Whisper transcription speed (about 9× real time on CPU), insert speed, memory over a session. It's as useful as the problem list, because it tells future me where not to spend a weekend.

Takeaways

• Measure your noise floor by comparing code with itself, before trusting any comparison.
• Generate every number in your docs from stored runs.
• Put regressions at the top of every comparison, and make accepting one a written decision.
• Check payload sizes next to timings. A fast response may just be a fast error.