Lorekeeper

Retrieval

lore search ranks passages, not files. It takes several wordings of one question in one call, runs offline, and never claims an answer is missing.

How a search works in Lorekeeper A four-step pipeline. One: capture. A thought or a link is filed by `lore capture` into your Markdown files, which stay plain and readable with no tool installed. Two: index. Every search builds an index in memory and then drops it, writing nothing to your notes; what it indexes is spans, a heading's passage addressed by file, anchor and line range, such as notes/consumer-leases.md#A lease is lost silently:15-22. Three: search. A coding agent asks one question three ways in a single call; each wording produces its own ranked list; the ranked lists are fused into one, where ranks vote together rather than scores, and near-duplicate suppression collapses copies of one passage to a single line. Four: answer. One result comes back in full — path, anchor, line range, score, and the span text itself — and that span text, rather than the whole notes it came from, is what reaches the agent's context. Version 0.1 makes no model call and no network call, and a score ranks passages against each other; it cannot prove a thing is absent. How a search works One capture, then every search rebuilds the index, ranks passages, and hands the agent span text. 1 · CAPTURE A thought, a link, something worth keeping lore capture Your Markdown files notes/consumer-leases.md Plain files you own, readable with no tool installed, this one included. 2 · INDEX Every search builds the index in memory, then drops it. Your notes are never written. Spans — a heading's passage, addressed notes/nightly-export.md#When it runs:16-18 notes/consumer-leases.md#A lease is lost silently:15-22 notes/oncall-triage.md#A run that reported success:20-22 A span carries its file, its anchor, and its line range. 3 · SEARCH Your coding agent asks one question three ways, in one call. "nightly export failed" "run stopped partway…" "consumer lease renewal…" Fused into one ranked list Near-duplicate suppression A span every wording found rises. Copies of one passage collapse to one line. 4 · ANSWER One result, in full notes/consumer-leases.md#A lease is lost silently lines 15-22 · score 0.032 · and the span text The agent's context The passages that answer the question, not the whole notes they came from. v0.1 makes no model call and no network call. A score ranks passages against each other; it cannot prove a thing is absent.

Spans

Every Markdown file is cut into spans. Headings come first, because a heading is where the author already decided one idea ends and the next begins. A section longer than 40 lines is then divided at a blank line, so no span is longer than one screen.

Each result is a span, with five fields: path, anchor (the heading it sits under, or none), startLine and endLine (counted from the first byte of the file, frontmatter included, so they match your editor's gutter), score, and text.

Nothing is derived from a path. Meaning is read from the text, which is why you can reorganize the folders without breaking retrieval.

Several wordings

Lexical search, one wording at a time, is not good enough, and this project measured that rather than assuming it. The same question asked in other words reaches notes the first wording could not. The examples below search the eight invented notes from the agent demo.

One wording.
lore search vault "nightly export failed" --limit 3
notes/nightly-export.md#When it runs:16-18  score 0.0164
~/vault/notes/nightly-export.md
    The nightly export starts at 02:10 and usually finishes before 02:40. It is
    scheduled rather than triggered, so a failed export does not retry on its own
    and waits for the next night unless somebody starts it by hand.

notes/nightly-export.md#Restarting a failed export:28-30  score 0.0161
~/vault/notes/nightly-export.md
    Starting the nightly export by hand is safe. The writer is keyed on the record
    identifier, so an export that runs twice lands the same rows twice and the
    warehouse keeps one of them.

notes/nightly-export.md#What it writes:22-24  score 0.0159
~/vault/notes/nightly-export.md
    Each nightly export writes one file per topic, named for the topic and the
    date, into the warehouse bucket. An export that produced no file for a topic
    either had nothing to move or did not finish.
Three wordings, one call.
lore search vault "nightly export failed" "run stopped partway and reported success" "consumer lease renewal committed offset" --limit 3
notes/consumer-leases.md#A lease is lost silently:15-22  score 0.0323
~/vault/notes/consumer-leases.md
    The renewal interval is thirty seconds. A consumer that spends longer than that
    inside a single batch never gets to renew, and the broker hands the partition
    to somebody else while the first consumer is still working.

    Nothing raises an error at that moment. The consumer finishes its batch and
    tries to commit an offset it no longer owns, the commit is refused, and the run
    stops at the last offset it did commit. From the outside the job simply stopped
    partway through and reported success.

notes/consumer-leases.md#How to tell this is what happened:26-28  score 0.0315
~/vault/notes/consumer-leases.md
    Compare the committed offset with the topic's end offset. A run that ended
    cleanly has caught up. A run that lost its lease is short by exactly the batch
    it was working on when the renewal was missed.

notes/oncall-triage.md#A run that reported success:20-22  score 0.0315
~/vault/notes/oncall-triage.md
    A run that reported success and still left the warehouse short did not fail in
    the writer. Look at what the consumers committed rather than at what the writer
    uploaded.

Each wording is ranked on its own, with BM25 (k1 1.2, b 0.75) over the span's text, plus a second BM25 over its metadata, weighted at 0.6. The metadata is the title, the heading trail, the anchor, tags, aliases, about, and the folder. The top 20 of each list are then fused by reciprocal rank fusion with k = 60. A span that repeats the same text in several files is reported once. Every one of these numbers was fixed before the evaluation ran and has never been tuned against a failing question.

JSON

--json prints the same results in a machine-readable shape. It is the retrieval contract an agent reads, and it does not change to make a number look better.

lore search vault "nightly export failed" "run stopped partway and reported success" "consumer lease renewal committed offset" --limit 1 --json
{
  "query": "nightly export failed | run stopped partway and reported success | consumer lease renewal committed offset",
  "queries": [
    "nightly export failed",
    "run stopped partway and reported success",
    "consumer lease renewal committed offset"
  ],
  "brain": "~/vault",
  "exhausted": false,
  "count": 1,
  "results": [
    {
      "path": "notes/consumer-leases.md",
      "anchor": "A lease is lost silently",
      "startLine": 15,
      "endLine": 22,
      "score": 0.03225806451612903,
      "text": "The renewal interval is thirty seconds. A consumer that spends longer than that\ninside a single batch never gets to renew, and the broker hands the partition\nto somebody else while the first consumer is still working.\n\nNothing raises an error at that moment. The consumer finishes its batch and\ntries to commit an offset it no longer owns, the commit is refused, and the run\nstops at the last offset it did commit. From the outside the job simply stopped\npartway through and reported success."
    }
  ]
}

score is the fused reciprocal-rank score that ordered the list, not a BM25 value. queries lists the wordings as given. exhausted is true when the brain holds more files than one search reads, so the ranking covers only part of it. A ranking over part of a vault is a different claim from a ranking over all of it, so the flag is reported, never swallowed. At a terminal, the same condition is written to standard error.

What a score cannot say

A score ranks spans against each other. It does not separate knowledge that is present from knowledge that is missing, so there is no relevance threshold: a cutoff would turn an honest miss into false confidence. When the results do not settle a question, search again in other words. When they still do not, say that the evidence is insufficient. Never report that the brain lacks something because a score looked low.

Measured: the context saving

Source: Feature 07, context/features/07-agent-integration-artifact.md, ## Verification. Ticket 07.2.

Measured: 2026-09-17, by npm run bench:context, from a clean build.

Corpus: 3,000 deterministic synthetic notes at seed 20260917, 3,193,061 bytes in total, averaging 1,064 bytes each, generated into a temporary directory and removed. Nothing real, nothing private, nothing committed.

Machine: Apple M5, macOS 25.6.0 arm64, Node v26.5.0.

The call measured is the one AGENTS.md teaches an agent to make: three quoted wordings, --json, default limit. The wordings come from the generator’s own vocabulary, because the synthetic corpus holds no content for the artifact’s illustrative TLS-certificate example to find. What was measured is the shape of the call — several wordings, fused — not those particular words.

Measure Prescribed call Sensitivity check, --limit 20
Results 5 20
Distinct files behind them 5 18
Whole notes, baseline 5,604 bytes 21,034 bytes
JSON payload received 2,847 bytes 9,985 bytes
Span text within it 1,564 bytes 5,913 bytes
Saving, payload against whole notes 1.97x 2.11x
Saving, span text against whole notes 3.6x 3.6x

The average returned span is 313 bytes.

The headline, and its ceiling

Approximately 2.0x, rounded up from the measured 1.97x. That is the ceiling on what is published for the prescribed call — not a claim about the best case, which the generous-baseline caveat below addresses — and it is the JSON payload against the baseline, because the payload is what a calling agent actually receives.

Span text is recorded beside it as a diagnostic, not as the headline. It is the evidence without the addressing that carries it, and no agent ever receives one without the other.

npm run bench:context prints ratios to one decimal, so the measured 1.97x appears as 2.0x in its own output, and 2.11x as 2.1x.

What the number does not claim

  • Bytes are a proxy for token cost, not a literal token count. No tokenizer and no provider coupling was added to produce these numbers. Anyone quoting this as a token saving is quoting something that was not measured.
  • The baseline is deliberately generous. It charges only for the files the returned spans came from, as though the agent had already known which files to open. Without span retrieval it would not have known — so 1.97x is a floor for the realistic case rather than a best case.
  • The corpus was not lengthened to flatter the result. Synthetic notes average roughly a kilobyte, so a span can only be about 3.4 times smaller than the note holding it. A vault of longer notes would show a larger saving, and the corpus was left as it was rather than tuned toward one.
  • The payload is pretty-printed, which is 12% of its bytes. Compact JSON would read 2.2x. The output format is a 05.1 contract and was not changed to improve this number.

The saving is real and modest, and it is reported as measured rather than as improved.

Frombrand/proof.md · The context saving

Measured: retrieval performance

Source: Feature 05, context/features/05-span-index-and-search.md, ## Verification. Ticket 05.3.

Measured: 2026-09-17, by npm run bench:search, from a clean build, over a deterministic synthetic corpus the command generates into a temporary directory and removes. Nothing generated is committed and nothing is read from a real vault.

Measure Value
Notes 3,000 (seed 20260917)
Spans 10,537
Corpus 3,193,061 bytes; 907 paragraphs duplicated across files
Walk + read + split 81 ms
Build index 64 ms
Query, one wording 2.3 ms median of 20
Query, three wordings fused and suppressed 6.9 ms median of 20
lore search end to end, three wordings, --json 129 ms median of 20
Machine Apple M5, 24 GiB, macOS 25.6.0 arm64, Node v26.5.0

The honest headline is the last row: a fused three-wording search over 3,000 notes completes end to end in about 130 ms, including building the whole in-memory index from scratch, because there is no index file and none is kept between commands.

What the number does not claim

  • “Interactive” is an interpretive reading, not an acceptance threshold. Feature 05 read the overview’s performance target as a fused three-wording search, index build included, finishing inside two seconds — the round trip an agent tool call tolerates. That reading was recorded as a reading. The measured result sits roughly fifteen times inside it.
  • One machine, one corpus, one shape of note. These are medians of twenty runs on an Apple M5 over generated notes, not a survey of real vaults on real hardware.
  • Building the index every time is a measured decision, not an oversight. At this size, persisting it would save roughly 150 ms and cost a manifest-owned file. Revisit only on a corpus where the walk and index phases, which scale with the vault, approach the reading.

Frombrand/proof.md · Retrieval performance

What has not been measured

Recorded so that the gaps are visible rather than inferred.

  • No measurement over a mature personal second brain. This is a recorded evidence boundary: the retrieval prototype was validated over substantial real-world Markdown project corpora, not over a populated personal brain. The real vaults available at the time were far too small to exercise ranking and were rejected for that reason, so the real-vault evidence base behind this project is very small and supports no quantitative claim at all. Proxy corpora under-represent note-to-note conceptual linking and dictated fragments.
  • No token-level measurement. See the proxy caveat above.
  • No measurement of answer quality end to end, with a real agent, on real questions. What is measured is what the retrieval layer returns and what it costs to return it.

Frombrand/proof.md · What has not been measured