It starts with "we use pnpm from now on" — how it gets captured, gated, stored, consolidated, recalled, injected, and how it ages out and says goodbye. Every station is a mechanism that really runs in the source; four of them are running live in your browser right now.
Cairn (@chenhw7/dsh-memory) is not a static table — it is a loop with inflow, outflow, and decay.
Under the hood it is eight Cordis rows: memory-store opens the storage domain and registers the ctx.memory service; tool-memory exposes the nine memory tools plus the identity_update identity tool; memory-review runs auto-extraction and consolidation; memory-notes renders conventions and pitfalls into the project-notes section (writing nothing into your repository); memory-identity seeds and reads the identity documents (SOUL.md / USER.md, off by default); memory-context owns system-prompt injection; memory-remote backs the memory console in Settings (remote writes denied by default); memory-root only discovers client modules. All rows share one storage domain.
The data lands in $DSH_HOME/storages/memory.json (the default host-medium backend) or $DSH_HOME/storages/memory.db (the SQLite backend) — uninstalling the plugin never deletes your memories.
One npm package carries all eight rows: host services are declared as peer dependencies, provided by the DeepSeek Harness you already have — never re-installed. The retrieval, tokenization, and scanning kernels are hand-rolled with zero third-party dependencies (a recorded engineering constraint), and SQLite rides Node's built-in node:sqlite. The tarball ships prebuilt, so the install path runs no build scripts — the smaller the supply chain, the smaller the trusted surface.
Every user message and tool result streams through a zero-LLM-cost projection accumulator — it only recognizes three kinds of signal.
The user asks to remember. Patterns like "记住 / 以后都 / remember that / from now on" hit and enter the candidate pool.
The user overturns an earlier claim: "不对 / 其实是 / actually / I meant" — a correction means the old belief was wrong, and the new conclusion is worth keeping.
Same-signature tool calls fail consecutively (≥2 by default) and then succeed — that success is a verified workaround, distilled into a pitfall entry. One-shot failures never enter.
Two flush paths keep signals from being lost: compaction end (compaction/end) extracts from the shadowed raw events; session disposal (session/disposed) makes one final pass over the whole session (5-second cap, fire-and-forget, never blocking the event).
The extraction prompt is itself a gate: anything the repository already records, transient states and unverified assumptions, procedures never verified by a tool are excluded; preferences need an explicit ask or a recurring topic. The line protocol emits [anchors:…] and [project] tags, with projectName taken from the highest-precedence source. With the identity layer on, a mechanical anti-echo prefilter also runs: candidates whose IDF-weighted overlap with the injected identity documents exceeds 0.6 are dropped — the documents' own echo never gets copied into the store.
Every 20 sessions, the curator picks the longest entries (≥400 characters) — at most 5 per batch — and has the extraction model rewrite them into concise one-liners. The store forgets, and it also distills.
The collection layer deliberately widens the funnel and pushes conservatism into the extraction prompt: a missed signal is a white loss — the truth is gone for good; a false positive is nearly free — the next station's dedup and consolidation will cut it. Put the gate in the wrong place and the costs are asymmetric.
Every write — from a model tool or background extraction — passes the same gate. Try it yourself:
The main memory table. Writes serialize on the domain write chain; past the soft cap, eviction follows the use signal (pinned immune; lowest accessCount / lastRecalledAt first), and when everything is exempt the store may exceed the cap — healthy data is never hard-deleted.
Every successful change appends: action, source (tool / review / flush / ui / janitor), content preview. Who wrote it and when is forever queryable.
The review mode's pending queue. A proposal is not a memory — not injected, not searchable, not decayed; it waits for a human verdict (station 07).
System metadata: the SQLite migration watermark (preventing a second migration), whole-store sweep progress and cooldown, consolidation-pipeline bookkeeping.
The current record of each identity document (SOUL.md / USER.md): content, version, and seed version, keyed by kind.
Version snapshots of the identity documents: every rewrite keeps one; a revert can only restore inside the window, and never destroys history.
The last two tables serve a separate opt-in layer (identityEnabled). SOUL.md is the assistant's character; USER.md is its working profile of you — the assistant rewrites them whole through identity_update in conversation (budgets: soul 2000 / user 3000 chars), and every rewrite keeps a version snapshot. The Identity settings section stays read-only: review history, revert to any retained version, never edit directly. When enabled, both documents inject as their own prompt sections (station 05); the extraction pipeline's anti-echo prefilter keeps the documents' own echo out of the memory store (station 01).
Storage has two backends behind one config switch:
| Dimension | host-medium (default) | sqlite |
|---|---|---|
| File | $DSH_HOME/storages/memory.json | $DSH_HOME/storages/memory.db |
| Engine | Host storage-domain KV (isomorphic with the other plugins) | Node's built-in node:sqlite, WAL + busy_timeout, zero extra dependencies |
| Write path | Serialized on the domain write chain; every write republishes the whole file | one transaction per logical write (entries + audit committed atomically) |
| Write amplification A/B (226 logical writes) | 628 whole-file publishes (2.78/write) · ~62 MB cumulative · 2.2 s | zero whole-file publishes · 139 KB · 0.79 s (≈2.8×) |
| Switching | Set storage: 'sqlite' and boot performs a one-time migration, stamping the migratedToSqlite watermark; afterwards either side booting wrongly fails loud — never a silent fork | |
There is also cross-process single-writer detection: the storage backend assumes one writer per process — two DSH processes sharing a storage root would silently clobber each other. On every boot memory-store stamps an owner mark (pid + bootId) into the domain's global slot and re-reads it periodically: reading back a live stamp that isn't yours proves a concurrent publisher, reported immediately. It only detects, never locks; a dead pid or a cleanly-exited predecessor is not a live threat, so restarts don't false-positive.
host-medium rides the host storage domain — isomorphic with the other plugins, no extra files — and small write volumes work perfectly well on it. SQLite serves high-write scenarios (auto-extraction + sweeps + hit stamps), collapsing whole-file write amplification into one transaction per write. Migration happens exactly once and is one-way, and both sides fail loud — two coexisting backends would only produce silent divergence, a worse failure than slowness.
A new entry doesn't just pile into the store — it first confronts the memories already there.
consolidation: 'legacy-judge'.consolidation: 'legacy-judge' keeps the old path behind a switch.A whole-store sweep (opt-in, off by default): every 20 session creations (plus one startup pass on the first session), the top 20 entries by use signal (hitCount / accessCount / recency) are paired exhaustively (candidate pairs capped at 20) and go through the verdict protocol again. Pairing still passes the lexical gate — overlap > 0.2, so pairs sharing zero tokens are filtered all the same; the sweep's real gain is the existing-vs-existing pairing surface and the use-signal ranking. Progress and cooldown (≥1 hour) persist in the meta table.
A usage-hit signal (opt-in, off by default): entries injected each round enter a per-session ledger; when the assistant's answer echoes an entry's tokens or anchors, hitCount / lastHitAt +1 — it only feeds sweep selection, never deletion.
An overturned conclusion that keeps being injected as a "fact" is the memory store's most dangerous lie — the model would argue with present reality using expired information. Superseding is not deleting: the original text, the audit trail, and the supersededBy pointer all survive, the evidence chain stays complete — it just stops being treated as current truth.
No vector store, no embedding model — a hand-rolled Okapi BM25 with CJK-aware tokenization and conservative Latin stemming.
Retrieval is two steps: structured filters first (scope / category / projectName), then BM25 scores the survivors — with document frequencies computed over the full corpus, so common words can't inflate inside a small candidate pool. The tokenizer splits Latin words and strips one conservative suffix to fold forms (uses→use, studies→study; class and basis are never damaged); for CJK it emits single-character unigrams plus adjacent bigrams — bigrams stop "记忆" from matching every entry containing "记", unigrams keep single-character queries alive. summary joins the index alongside content. Final order: relevance ↓ → pinned ↓ → importance (absent reads 0, losing ties) ↓ → recency ↓.
Zero-hit fallback: a non-empty query with no lexical hit returns the most recent entries under the same filters, flagged fallback: true — in the tool return only; injection surfaces never fall back. The benchmark exports as @chenhw7/dsh-memory/benchmark, and the golden set carries CI floors: success@5 ≥ 0.85, MRR ≥ 0.75, P@1 ≥ 0.6, zh-slice success@5 ≥ 0.8.
Zero dependency is a recorded engineering constraint: BM25 is explainable, free at runtime, and runs offline in tests; Chinese bigram tokenization restores word-level precision. The honest cost is stated on the table — no cross-lingual or paraphrase semantics; the golden set's synonym slice rides on summary indexing. When real workloads prove lexical isn't enough, evaluate the upgrade then — today, don't pay complexity for a hypothetical requirement.
How memory enters the context decides every token it saves the model.
Each session freezes a memory snapshot at creation and reuses it for the whole session — prefix unchanged, KV-cache fully warm. The only re-freeze is compaction (the prompt is rebuilt anyway), so memories learned mid-session slip in exactly there. The memory section sits at system-prompt order 6000 (right after the host's tool-guidance section at 5000), with project-notes right behind at 6001; with the identity layer on, the soul and user-profile sections sit further ahead at 80 / 81.
Optional step-level auto-recall takes the other road: on every agent step, a BM25 search keyed on the step's user text appends matching entries as a <recalled-memory> fenced user message — max 5 entries, queries at least 12 characters, fence ≤1200 chars with a token footer; the system prompt is untouched, the prefix cache unmoved. At load time, entries contradicted by newer facts carry conflict annotations (fresh / stale / conflicting, from a pure IDF-weighted detector), letting the model weigh old against new itself.
Standing cost should decouple from store size: policy-only is a flat ≈344 tokens, and digest adds only a one-time ≈126-token category-and-topics inventory on the first step — the model knows roughly what the store holds and pulls full text via memory_search / memory_get on demand. No entry data enters the system prompt at all, so the KV prefix stays perfectly stable; stores that want a stronger presence can still switch to index or full (dual budget: 5000 chars / 20 entries).
The janitor patrols at every session creation. Drag the timeline — watch three entries meet different fates:
"Deploy script lives in scripts/deploy.sh, port 8080"
"Behind a proxy, pnpm needs the registry mirror set first"
"Code-review comments must each carry a reason"
The project scope is bound to a repository: the repo moves on, the conventions change — expired project memories left in place only pollute retrieval, so they are hard-deleted when overdue. global / user are cross-project assets where a wrong deletion costs far more than retention — stamp instead of delete, and one recall overturns the stamp. Different stakes deserve different deaths.
Deletion is the heaviest operation in the system — so every step of it is gated.
An explicit confirm: true double confirmation; batches over half the search cap are refused outright (runaway protection); pinned entries are skipped and reported; every removal is individually audited. Topic-level forgetting never sweeps away your pinned keepers.
Scanner-rejected content never lands; content already stored that turns out problematic comes back through memory_get's raw: true for repair — and the break-glass read itself enters the audit. The original is always inspectable; silent deletion would only hide the attack.
Finally, the remote boundary: the memory console's 18 RPC methods are read-only by default — 10 reads and 8 writes, the writes sitting behind one deployment-level switch (remoteWritesEnabled); identity governance owns three methods of its own (identityList / identityHistory / identityRevert), and a revert must also pass a second switch (identityRevertEnabled). What is trusted is the transport fence (loopback / LAN / trusted hosts, owned by the host), not the storage face: a wider trust scope never silently grows into whole-store write access.
Every metric on this page has a source — either CI-gated, or measured against the real DeepSeek Harness.
| Layer | What it measures | How it runs |
|---|---|---|
| L0 | golden-set retrieval benchmark (35 entries × 35 queries, en / zh / mixed, with synonym and Latin-inflection slices) | CI gates: success@5 ≥ 0.85 · MRR ≥ 0.75 · P@1 ≥ 0.6 · zh-slice success@5 ≥ 0.8 |
| L1 | behavioral eval: core-v0, 32 scenarios / 132 questions + noise-v0, 6 long-noise scenarios + identity-v0, 3 scenarios / 6 questions | runs the real harness with a deterministic mock model — no credentials, green on every push |
| L2 | real-model judged scoring, versioned rubrics (storage-v2 / recall-v2) | environment-gated; the judge is deliberately never the model under test — no self-grading bias |
| Unit | 1096 test cases (1090 passing + 6 key-gated skips) | the store contract runs against three backends: in-memory / JSON domain / SQLite |
Proposals observed repeatedly accumulate hits and float to the top; entries recalled repeatedly stay fresh. Usage traces are the most valuable metadata there is.
The snapshot freezes for the KV-cache for the whole session — better a one-beat delay than a wobbling prefix. Cache hits save far more than instant injection earns.
Every injection surface declares it: the current user request, repository files, and tool output always outrank memory. On conflict, trust the present.
37 scanner rules, audit / suggestions rolling at 200 each, entries soft-capped at 500, the optional review queue — a memory system must be able to answer "who wrote this memory, when, and why".