MindSage engineering notes · 02

Vector search was 0.03% of my chat pipeline

My local "ask your journal" feature took 14 seconds to answer. I assumed retrieval was the hard part. The benchmark said retrieval was basically free, and two language model calls were the whole bill.

By Arinay · November 2026 · About 7 minutes

MindSage has a feature called Ask. You type a question like "when did my knee start hurting?", and a local model answers from your own journal entries. Everything runs on your PC: a 3B Llama 3.2 model through Ollama for text, nomic-embed-text for embeddings, and a vector index inside the app's SQLite file.

It worked, but it was slow. So I timed each stage separately instead of guessing.

The pipeline

A question goes through four stages, one after another:

1. Query planning. The model reads the question and returns JSON: does it need journal context, does it mention a time range, and what's the core search phrase.
2. Embedding. The question becomes a 768-number vector.
3. Vector search. The five closest entries come back.
4. Answer generation. The model gets the question plus those five entries and writes the answer.

Stages 2 and 3 look like the clever engineering. That's where I expected the time to go.

Where the 14 seconds went

Retrieval, embedding and search together, was a fifth of one percent. The whole cost was two language model calls in a row, against an Ollama server that handles one request at a time. Nothing I could do to the vector index would ever show up in this number.

That result later justified a decision I wrote about in post 01: replacing the Qdrant search server with sqlite-vec, which is slower at search but removes a whole process. Making a 4.5 ms step 20 ms slower changes nothing a user can feel.

The same shape showed up twice

Saving a journal entry had the same problem. The app asks the model for metadata (title, mood score, mood tags) and then, separately, for a summary:

Both features make two generations, and in both the first one only exists to produce structured data for the second. That's a design smell I didn't see until the numbers sat side by side.

One more number from the same run: the first AI call after launch took 6.40 s, of which 6.21 s was loading the model from disk. The same call warm took 224 ms. Whatever feature the user touches first pays that 28× penalty.

What I changed first: stream the answer

The quickest win doesn't make the pipeline shorter. It makes the wait feel shorter. On its own, the model's time to first token was about 400 ms. So if the answer streams into the window as it's written, the user waits for planning plus that first token, then reads while the rest arrives.

There was a catch. The answer step asks the model for JSON, so the response comes back as {"response": "…"}. You can't show raw JSON to a user, and you can't parse a half-finished JSON object either. So the stream handler pulls the partial response string out of the incomplete JSON on every token and sends only the new characters:

onToken: (_fragment, full) => {
  const partial = partialJsonString(full, "response");
  if (partial === null || partial.length <= emitted) return;
  onDelta(partial.slice(emitted));
  emitted = partial.length;
},

If the JSON fails validation, the app retries the call. A failed attempt may already have streamed half an answer, so before each retry the renderer gets a reset event and throws the partial text away. Without that, the second attempt appended itself to the first.

I haven't benchmarked "time until the first word appears" on the full pipeline yet, so I'm not quoting a number for it. That's the next metric to add.

Where it stands

The latest full run, on 20 October, measured planning at 4.13 s, embedding at 20 ms, search at 1.2 ms and the answer at 6.32 s, for 10.48 s p50 in total. That's down from 14.06 s, but I can't pin the drop on one change. LLM timings on a laptop move a lot between runs, so I treat it as "somewhat faster", not as a result.

The real fixes are still ahead, in this order:

1. Skip planning when it can't help. Most questions either obviously need the journal or obviously mention a date. A cheap check can handle those and save the model call for unclear ones. That's up to 4 seconds off most questions.
2. Send smaller context. Five whole entries go into the answer prompt. Summaries would cut the prompt reading time.
3. Pre-warm the model after launch, so nobody pays the 6-second cold load.
4. Fix retrieval quality. The same benchmark suite showed the top search hit was wrong more than half the time. So those 14 seconds were often spent reasoning over the wrong entries. That's post 03.

Takeaways

• Time every stage of an AI pipeline separately before optimising any of it. The part that looks hard is rarely the part that's slow.
• On local hardware the model call is the unit of cost. Count model calls before you count milliseconds.
• When a feature can't get faster quickly, streaming changes how long it feels. But measure the time to the first word, or you're only guessing.