Side B makes concrete, working improvements: it upgrades the dev fixture script to use cargo-watch for live-reload, persist fixture data across runs (rebase/reuse summary), prefer a stable port, and clean up UI markup in forum.rs — all directly usable, testable code changes. Side A is purely a planning document (PLAN.md) with no code changes; while it may guide future work, it has no immediate functional impact and its value depends entirely on future execution, making B's tangible, working tooling improvement more durable/immediately valuable.
constitution · epochs · watch · epoch 3
comparison
c_6c64824b0d83 (tommy-mor) vs c_a337088f57f0 (tommy-mor)
download prompt · raw event · cmp_d0ad808559fbc9
council reasoning
A adds a concrete, phased storage/architecture plan (JSONL as source of truth, durable RocksDB projections, RAM bounded to hot scopes, write/read/startup paths) that defines lasting product direction under the 256MB constraint. B only improves local DX (cargo-watch fixture, persistent fixture-data/, preferred port) plus a small room-page markup trim—useful iteration friction reduction, but not fundamental system design.
Side B makes functional improvements to the developer workflow by switching the fixture runner to cargo-watch with persistent fixture data, adding preferred-port selection with fallback, reusing seeded data across runs, extending server startup timeouts, and updating supporting utilities. Side A is an extensive design document describing a future storage architecture and implementation plan, but it does not change project behavior or implement the proposed design.
sides
A — c_6c64824b0d83 (tommy-mor)
message
[a2a83d76] plan
diff preview
diff --git a/PLAN.md b/PLAN.md
new file mode 100644
index 0000000000000000000000000000000000000000..38291545caebde2632e3fe409ea54c63abc30f5a
--- /dev/null
+++ b/PLAN.md
@@ -0,0 +1,337 @@
+# sorter2 storage & memory plan
+
+## Goal
+
+Fit a **decent chunk of Reddit** into a **256MB** Fly VM while keeping the product simple: one Rust binary, no external database service.
+
+**RAM should be bounded by query shape**, not dataset size — ideally one rank-centrality graph in memory at a time, plus runtime overhead.
+
+**`events.jsonl` remains the main database.** Everything on disk elsewhere is a **rebuildable projection**.
+
+---
+
+## Architecture (target)
+
+```
+ ┌─────────────────────────────────┐
+ │ events.jsonl (source of truth) │
+ └───────────────┬─────────────────┘
+ │
+ append on every mutation
+ │
+ ▼
+ ┌──────────────────────────────────────────────┐
+ │ apply event → durable projection (on disk) │
+ │ (same semantics as today's in-memory reducer) │
+ └──────────────────────────────────────────────┘
+ │ │
+ ▼ ▼
+ ┌──────────────────┐ ┌──────────────────────────┐
+ │ entity_payloads │ │ reducer state per scope │
+ │ (fat Reddit JSON)│ │ nodes, children, edges, … │
+ └──────────────────┘ └──────────────────────────┘
+ │ │
+ └────────┬───────────┘
+ ▼
+ ┌──────────────────────────────────────────────┐
+ │ RAM per request (or small LRU cache) │
+ │ • one scope's GroupState for rank-centrality │
+ │ • children + EntityData (small views) │
+ │ • RC scratch allocations │
+ │ → compute → render → drop / evict │
+ └──────────────────────────────────────────────┘
+```
+
+This is **event sourcing / CQRS**:
+
+| Layer | Role |
+|-------|------|
+| **JSONL** | Canonical write log; audit; disaster recovery |
+| **Durable (RocksDB)** | Materialized read model + payload store; rebuildable from JSONL |
+| **RAM** | One (or few) hot scopes for ranking and render |
+
+Durable is **not** a second source of truth. If projection and log diverge, **stream JSONL and rebuild durable**.
+
+---
+
+## What we have today (baseline)
+
+| Piece | Status |
+|-------|--------|
+| `events.jsonl` append-only log | ✓ source of truth |
+| `EventLog::replay` streaming one event at a time | ✓ no full `Vec<Event>` at startup |
+| `entity_store` / `entity_db` (RocksDB via `durable`) | ✓ fat payloads off-heap |
+| `EntityData` in `GlobalTree` | ✓ small derived views in RAM |
+| Full `GlobalTree` replayed at boot | ✗ all nodes, all scopes' `GroupState` in RAM |
+| Rank-centrality | ✓ already scoped per parent; reads in-memory `GroupState` |
+
+**Measured RSS (release, ~395 imports, 2.8MB JSONL):**
+
+| Scenario | RSS |
+|----------|-----|
+| Empty data dir | ~14 MB (+ RocksDB baseline) |
+| After boot with data | ~21–22 MB |
+| Pre-offload (in-tree payloads + vec replay) | ~25–34 MB |
+
+Payload offload + streaming replay helped startup peak, but **the full in-memory reducer** is still the scaling ceiling.
+
+---
+
+## What lives where (target)
+
+### JSONL (`events.jsonl`)
+
+All mutations, append-only:
+
+- `VoteRecorded` — scope, pair, ratios
+- `EntityImported` — id, full upstream payload
+- `NodeEnsured` — register path
+- (legacy / other event types as present in log)
+
+### Durable / RocksDB (`{data_dir}/…`)
+
+Single embedded DB directory. Collections (names tentative):
+
+| Collection | Contents | Notes |
+|------------|----------|-------|
+| `entity_payloads` | `ItemId → JSON string` | **Done.** Fat Reddit API blobs |
+| `nodes` | `ItemId → { data: EntityData, children: … }` | Small; no raw payload |
+| `scopes/{parent}/…` | `GroupState` materialization | edges, voted_pairs, item_to_idx, recent_votes (capped) |
+
+Nested layout can follow durable's `Map → Map → Vec` patterns (see `durable/docs/001.md` Sorter sketch).
+
+### RAM
+
+| Resident | When |
+|----------|------|
+| Tokio, Axum, reqwest, RocksDB block cache (tuned) | always |
+| **One scope slice** | per request (or LRU of few scopes, byte-capped) |
+| Rank-centrality temporaries | during render for that scope |
+
+**Not** in RAM at steady state: all subreddits, all vote graphs, all payloads.
+
+---
+
+## Write path
+
+Order matters:
+
+1. Append event to `events.jsonl` (must durably succeed first)
+2. Apply event to durable projection (same logic as today's `apply_event`)
+3. Invalidate / update in-memory scope cache if that scope is hot
+
+Journal worker already serializes votes disk → tree; extend to **disk → durable** instead of (eventually) **disk → full GlobalTree**.
+
+```rust
+// conceptual
+append(jsonl, event)?;
+apply_to_durable(event)?;
+scope_cache.invalidate(scope_for(event));
+```
+
+On failure after (1): replay from log repairs projection on next boot or via `replay-index` command.
+
+---
+
+## Read path
+
+For a page under parent scope `P` (e.g. `reddit.com/r/rust`):
+
+1. **Load scope** from durable (or scope LRU hit)
+ - `EntityData` + children for listing
+ - `GroupState` for ranking and pair selection
+2. **Run rank-centrality** on that `GroupState` (requires RAM — that's fine)
+3. **Render**
+4. **Drop** scope from RAM or return to LRU
+
+Payload fetch (rare): `entity_store.get(id)` only when render needs fields not in `EntityData`.
+
+---
+
+## Startup & recovery
+
+### Normal startup
+
+```
+open entity_db (RocksDB)
+open / validate scope indexes in same DB
+do NOT replay JSONL into RAM
+serve requests (cold scopes loaded on demand)
+```
+
+### Rebuild projection
+
+```
+stream events.jsonl → apply_event → durable
+(one line at a time; same as EventLog::replay today)
+```
+
+Run when:
+
+- First deploy of projection layer
+- Detected corruption / missing durable dir
+- Manual `replay-index` after restoring JSONL from backup
+
+JSONL is the only file you need to trust for recovery.
+
+---
+
+## Scope cache (RAM bound)
+
+**Strict mode:** one scope in RAM at a time — simplest, lowest RAM.
+
+**Practical mode:** LRU cache with **byte budget** (e.g. 64–128MB for scopes on a 256MB VM):
+
+- Evict least-recently-used scope's `GroupState` + child views
+- Reload from durable on next visit
+
+Eviction policy is independent of storage engine.
+
+---
+
+## Rank-centrality
+
+No change to the algorithm. It already assumes a whole `GroupState` for one parent scope.
+
+Moving reducer to durable does **not** remove RC memory cost — it removes **holding every scope's graph at once**.
+
+Optional later: materialized score vectors on disk, invalidated on vote. Not required for v1 of this plan.
+
+---
+
+## Durable mutations (future API)
+
+Separate **intent** from **apply** for batching and testability:
+
+```rust
+let m = rankings.path().key("rust").key(day).push_end(score);
+db.apply(m)?; // or batch.apply(&[m1, m2, m3])
+```
+
+Benefits:
+
+- One RocksDB `WriteBatch` / one WAL flush per vote or import batch
+- Serializable ops for tests
+- Aligns with JSONL events at the app layer and storage ops at the durable layer
+
+Keep chained `entry().push()` as sugar over `path().…; apply()`.
+
+Type safety: use type-state path builders if we want compile-time nesting; erased `Vec<Op>` only if we accept runtime errors at apply.
+
+---
+
+## Storage engine: RocksDB vs alternatives
+
+**Current choice:** RocksDB via vendored `durable/` workspace crate.
+
+| Engine | Verdict for sorter2 |
+|--------|---------------------|
+| **RocksDB** | Good default for LSM, prefix scans, write-heavy votes + bulk imports. C++ dep, tune block cache for 256MB. |
+| **sled** | Pure Rust appeal; production reliability history gives pause. Not a priority switch. |
+| **fjall** | Pure Rust LSM; evaluate with benchmarks if leaving RocksDB. |
+| **redb** | Lighter pure Rust; fine for payload-only store; less ideal for heavy scattered writes across scopes. |
+| **SQLite** | Wrong shape for nested fractal tree; OK for a single KV table only. |
+
+**Switching engines matters less than:**
+
+1. Batched durable writes
+2. Not materializing full reducer in RAM
+3. Scope-local load/evict
+
+RocksDB stays **narrow**: blob attic + materialized reducer projection. Not a replacement for JSONL.
+
+---
+
+## Scaling story (before vs after)
+
+| | In-memory reducer (today) | Target (JSONL + durable projection) |
+|--|---------------------------|-------------------------------------|
+| **RAM grows with** | Total nodes + all scopes + (was) payloads | Hot scope count × scope size + runtime |
+| **Disk grows with** | JSONL (+ entity_db payloads today) | JSONL + full durable projection |
+| **Startup** | O(events) replay into RAM | O(1) open DB |
+| **Fails when** | RSS > VM limit | Scope too large for one RC graph, or disk full |
+| **Recovery** | Replay JSONL | Replay JSONL → rebuild durable |
+
+**Rough RAM per active subreddit (~500 posts, moderate votes):**
+
+| Component | Order of magnitude |
+|-----------|-------------------|
+| `EntityData` × 500 | 0.5–2 MB |
+| `GroupState` | 1–5 MB |
+| RC scratch | 1–5 MB |
+| Runtime + tuned RocksDB | 15–25 MB |
+| **Total one hot scope** | **~20–40 MB** |
+
+Multiple subreddits fit on 256MB with LRU eviction, not all resident at once.
+
+---
+
+## Implementation phases
+
+### Phase 0 — Done
+
+- [x] Vend `durable` as workspace crate (`durable/`)
+- [x] `entity_store`: payloads in `{data_dir}/entity_db`
+- [x] Remove `entity_raw` from `NodeState`
+- [x] `EventLog::replay`: stream JSONL, one `Event` at a time
+- [x] `apply_event` in `state.rs` for replay semantics
+
+### Phase 1 — Durable projection (write path)
+
+- [ ] Single `Db` under `{data_dir}/store` (payloads + reducer)
+- [ ] `apply_event` writes to durable collections (nodes, scopes) in addition to or instead of `GlobalTree`
+- [ ] Journal + Reddit import paths use same apply
+- [ ] Batched writes where possible (one flush per vote / per import batch)
+- [ ] Tests: apply event → read back from durable
+
+### Phase 2 — Stop full tree at boot
+
+- [ ] Startup: open durable only; no `GlobalTree::new()` + full replay into RAM
+- [ ] `replay-index` command / flag: stream JSONL → durable (offline rebuild)
+- [ ] Integration tests use replay-index fixture or temp DB
+
+### Phase 3 — Scope load on read
+
+- [ ] `ScopeView { parent, children, group }` loaded from durable
+- [ ] HTML / vote / pair / fetch handlers take `ScopeView` instead of `&GlobalTree`
+- [ ] Remove or shrink `Arc<RwLock<GlobalTree>>`
+
+### Phase 4 — Scope cache
+
+- [ ] LRU with byte budget for hot scopes
+- [ ] Invalidate on write to that scope
+- [ ] Metrics: cache hit/miss, evictions, scope load time
+
+### Phase 5 — Durable API polish (optional)
+
+- [ ] Reified path/mutation API + `WriteBatch` integration in `durable`
+- [ ] RocksDB tuning preset for 256MB Fly (`block_cache`, write buffers)
+- [ ] Document `replay-index` in AGENTS.md
+
+---
+
+## Non-goals (for now)
+
+- Distributed replication or multi-writer
+- Replacing JSONL as canonical store
+- SQL query layer over state
+- Materialized rank scores on disk (unless RC latency forces it)
+- Switching from RocksDB to sled without benchmarks
+
+---
+
+## Open questions
+
+1. **One DB or two?** `{data_dir}/entity_db` today vs single `{data_dir}/store` — merge on Phase 1?
+2. **Scope key encoding** — string `ItemId` paths vs hashed; must match event `scope` field.
+3. **Child scope wiring** — Reddit `apply_entity_under_parent` creates children without full path ensure; durable schema must preserve this.
+4. **Clojure smoke tests** — still read `events.jsonl`; durable is internal. No change expected.
+5. **Fly volume** — `/data
… preview truncated; 478 characters omittedB — c_a337088f57f0 (tommy-mor)
message
[a45842df] better dev iteration script
diff preview
diff --git a/.gitignore b/.gitignore
index 9cd2eaece834481bee87639cd7cb6be77f33e302..b70cf5b97cd3556e061233d8b0aefd333568a991 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,6 +3,7 @@ target/
repomix-output.xml
psalms_kjv.txt
dev-data/
+fixture-data/
.worktrees/
.specstory/
.specstory/
diff --git a/bb.edn b/bb.edn
index 5d1d642d75d03f70879d7673e7ad95360336a8b1..4680e82d4227057d2eea6041dc5647beb759131d 100644
--- a/bb.edn
+++ b/bb.edn
@@ -47,7 +47,7 @@
"RUST_LOG" "info"})})))}
fixture
- {:doc "Run local server + mock OAuth + seeded walkthrough data for manual browser demos"
+ {:doc "Run local server via cargo-watch + mock OAuth + seeded walkthrough data (persistent ./fixture-data/, prefers PORT 8080). Requires cargo-watch."
:requires ([test.walkthrough-fixture :as walkthrough-fixture])
:task (walkthrough-fixture/run-fixture)}
diff --git a/server/src/html/forum.rs b/server/src/html/forum.rs
index e00e7110e2ad778d63b3d00b0cc4ffe20ba2e0d4..815fc6529062b103553644fc15e516334aec2b88 100644
--- a/server/src/html/forum.rs
+++ b/server/src/html/forum.rs
@@ -976,12 +976,6 @@ pub async fn room_page(
html! {
(strip)
nav class="breadcrumb" { (bc_room(&nav, slug_display, None)) }
- h2 { (slug_display) }
- p class="muted" { (room_id) }
- p class="muted room-links" {
- "room garden · "
- a href=(nav.garden_root_url()) { "~" }
- }
(room_members_section_markup(&reduced, &room_id, false))
h3 { "threads" }
(render_thread_feed(Some(&nav), "room-thread-feed", &rows, now))
diff --git a/test/common.clj b/test/common.clj
index 76a3e99fdb67c10bcb5c2f53d8bcd0917398af50..c6deba0d90995b0e3f9c4b8ad66a516ed19b870e 100644
--- a/test/common.clj
+++ b/test/common.clj
@@ -107,6 +107,16 @@
(.close ss)
port))
+(defn pick-port-prefer
+ "Use `preferred` if it can be bound, otherwise an ephemeral port (same as `pick-port`)."
+ [preferred]
+ (try
+ (let [ss (java.net.ServerSocket. preferred)]
+ (.close ss)
+ preferred)
+ (catch java.io.IOException _
+ (pick-port))))
+
(defn wait-for-server
"Poll /healthz until it returns 'ok', up to `timeout-ms`."
[base-url timeout-ms]
diff --git a/test/walkthrough_fixture.clj b/test/walkthrough_fixture.clj
index 4b814822029c913e38151ca53848d1c64fb58373..a487ff1ef8548580577914fb33f321586859c9af 100644
--- a/test/walkthrough_fixture.clj
+++ b/test/walkthrough_fixture.clj
@@ -1,11 +1,18 @@
(ns test.walkthrough-fixture
"Launch a local slug server with mock OAuth and seed browser-friendly demo data."
(:require [babashka.fs :as fs]
+ [babashka.process :as p]
[cheshire.core :as json]
[clojure.string :as str]
[test.common :as common]
[test.oauth :as oauth]))
+(def ^:private fixture-data-dir "fixture-data")
+;; Prefer the same port as `bb dev` / `bb watch`; fall back if something else is listening.
+(def ^:private preferred-slug-port 8080)
+;; First `cargo watch` compile can exceed a release-binary startup; allow several minutes.
+(def ^:private health-wait-ms 300000)
+
(defn- assert! [pred msg]
(when-not pred
(throw (ex-info msg {}))))
@@ -85,49 +92,87 @@
:thread_url (str base-url "/r/" room-short "/" room-slug "/t/walkthrough-thread")
:garden_url (str base-url "/r/" room-short "/" room-slug "/~/secret/item")}}))
+(defn- rebase-fixture-summary [saved current-base-url current-google-url current-data-dir]
+ (let [inner (:summary saved)
+ room (:room inner)
+ rs (:short room)
+ lg (:slug room)]
+ (assoc saved
+ :base_url current-base-url
+ :mock_google_url current-google-url
+ :data_dir (str current-data-dir)
+ :summary (assoc inner
+ :room (assoc room
+ :url (str current-base-url "/r/" rs "/" lg)
+ :thread_url (str current-base-url "/r/" rs "/" lg "/t/walkthrough-thread")
+ :garden_url (str current-base-url "/r/" rs "/" lg "/~/secret/item"))))))
+
+(defn- fixture-log-present? [data-dir]
+ (let [p (fs/path data-dir "events.jsonl")]
+ (and (fs/exists? p) (pos? (fs/size p)))))
+
+(defn- load-or-seed-summary!
+ [base-url google-url data-dir summary-path]
+ (if (and (fixture-log-present? data-dir) (fs/exists? summary-path))
+ (let [saved (json/parse-string (slurp summary-path) true)
+ rebased (rebase-fixture-summary saved base-url google-url data-dir)]
+ (spit summary-path (json/generate-string rebased {:pretty true}))
+ (println "")
+ (println "reusing fixture-data/ (delete the directory for a fresh seed)")
+ rebased)
+ (let [seeded (seed-demo! base-url)
+ s {:base_url base-url
+ :mock_google_url google-url
+ :data_dir (str data-dir)
+ :summary seeded}]
+ (spit summary-path (json/generate-string s {:pretty true}))
+ s)))
+
(defn run-fixture [& _args]
- (let [build (common/run-cargo-build-release! ["slugsocial-server"])
- _ (assert! (zero? (:exit build)) "cargo build --release failed")
- server-bin "target/release/slugsocial-server"
- tmp-dir (str (fs/create-temp-dir {:prefix "slug-walkthrough-"}))
- slug-port (common/pick-port)
+ (let [data-dir (str (fs/absolutize (fs/path (fs/cwd) fixture-data-dir)))
+ slug-port (common/pick-port-prefer preferred-slug-port)
google-port (common/pick-port)
base-url (str "http://127.0.0.1:" slug-port)
google-url (str "http://127.0.0.1:" google-port)
- stable-dir "/tmp/slug-walkthrough-fixture"
- summary-path (str stable-dir "/summary.json")
+ summary-path (str (fs/path data-dir "summary.json"))
!server (atom nil)
!google (atom nil)
- server-env (common/slug-server-env tmp-dir base-url google-url slug-port)]
+ server-env (merge (common/slug-server-env data-dir base-url google-url slug-port)
+ {"RUST_LOG" "info"})
+ watch-cmd [(common/cargo-bin) "watch"
+ "-x" "run -p slugsocial-server"
+ "-w" "server/src"
+ "-w" "server/static"
+ "-w" "types/src"]]
(try
- (fs/create-dirs stable-dir)
+ (fs/create-dirs data-dir)
(reset! !google
(oauth/start-mock-google google-port
:google-users ["google-user-alice" "google-user-bob"]))
- (reset! !server (common/start-server server-bin server-env))
- (assert! (common/wait-for-server base-url 10000) "server did not become healthy")
- (let [seeded (seed-demo! base-url)
- summary {:base_url base-url
- :mock_google_url google-url
- :data_dir tmp-dir
- :summary seeded}]
- (spit summary-path (json/generate-string summary {:pretty true}))
+ (println "")
+ (println "starting cargo-watch (first compile may take a while)…")
+ (flush)
+ (reset! !server (p/process watch-cmd {:inherit true :env server-env}))
+ (assert! (common/wait-for-server base-url health-wait-ms) "server did not become healthy")
+ (let [summary (load-or-seed-summary! base-url google-url data-dir summary-path)]
(println "")
(println "walkthrough fixture ready")
(println (str " base url: " base-url))
(println (str " room page: " (get-in summary [:summary :room :url])))
(println (str " thread page: " (get-in summary [:summary :room :thread_url])))
(println (str " garden page: " (get-in summary [:summary :room :garden_url])))
+ (println (str " data dir: " data-dir))
(println (str " summary json: " summary-path))
(println "")
(println "seeded users")
(println " alice / bob via mock OAuth")
(println "")
+ (println "editing server/src or server/static reloads the server; data persists in fixture-data/")
+ (println "")
(println "press Ctrl-C to stop")
(flush)
(while true
(Thread/sleep 1000)))
(finally
(when-some [s @!server] (common/kill-server s))
- (when-some [g @!google] ((:stop-fn g)))
- (fs/delete-tree tmp-dir)))))
+ (when-some [g @!google] ((:stop-fn g)))))))
Hardlinks — judgments / attempts / prompt
judgments
attempts
Prompt text is loaded only by the download route.