You are a constitutional council ranking individual git commits for ownership allocation. Compare these two commits. Decide which contributed more lasting value to the project. Judge substance, not spectacle: - Prefer correct, lasting design and real bugfixes over churn, formatting, renames, or generated noise. - Prefer clarity and necessity over sheer line count. A small precise change can beat a large diffuse one. - Do not favor a side merely because its patch is longer or noisier. - Weight what the change does for the project, not the contributor's name. Return ONLY a JSON object: {"winner": "A" or "B", "ratio": "N:M", "explanation": "..."} The explanation must cite concrete differences in the patches (1-3 sentences). Side A — contributor: tommy-mor Side A — commit message: [a2a83d76] plan Side A — unified diff (full patch): diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000000000000000000000000000000000000..38291545caebde2632e3fe409ea54c63abc30f5a --- /dev/null +++ b/PLAN.md @@ -0,0 +1,337 @@ +# sorter2 storage & memory plan + +## Goal + +Fit a **decent chunk of Reddit** into a **256MB** Fly VM while keeping the product simple: one Rust binary, no external database service. + +**RAM should be bounded by query shape**, not dataset size — ideally one rank-centrality graph in memory at a time, plus runtime overhead. + +**`events.jsonl` remains the main database.** Everything on disk elsewhere is a **rebuildable projection**. + +--- + +## Architecture (target) + +``` + ┌─────────────────────────────────┐ + │ events.jsonl (source of truth) │ + └───────────────┬─────────────────┘ + │ + append on every mutation + │ + ▼ + ┌──────────────────────────────────────────────┐ + │ apply event → durable projection (on disk) │ + │ (same semantics as today's in-memory reducer) │ + └──────────────────────────────────────────────┘ + │ │ + ▼ ▼ + ┌──────────────────┐ ┌──────────────────────────┐ + │ entity_payloads │ │ reducer state per scope │ + │ (fat Reddit JSON)│ │ nodes, children, edges, … │ + └──────────────────┘ └──────────────────────────┘ + │ │ + └────────┬───────────┘ + ▼ + ┌──────────────────────────────────────────────┐ + │ RAM per request (or small LRU cache) │ + │ • one scope's GroupState for rank-centrality │ + │ • children + EntityData (small views) │ + │ • RC scratch allocations │ + │ → compute → render → drop / evict │ + └──────────────────────────────────────────────┘ +``` + +This is **event sourcing / CQRS**: + +| Layer | Role | +|-------|------| +| **JSONL** | Canonical write log; audit; disaster recovery | +| **Durable (RocksDB)** | Materialized read model + payload store; rebuildable from JSONL | +| **RAM** | One (or few) hot scopes for ranking and render | + +Durable is **not** a second source of truth. If projection and log diverge, **stream JSONL and rebuild durable**. + +--- + +## What we have today (baseline) + +| Piece | Status | +|-------|--------| +| `events.jsonl` append-only log | ✓ source of truth | +| `EventLog::replay` streaming one event at a time | ✓ no full `Vec` at startup | +| `entity_store` / `entity_db` (RocksDB via `durable`) | ✓ fat payloads off-heap | +| `EntityData` in `GlobalTree` | ✓ small derived views in RAM | +| Full `GlobalTree` replayed at boot | ✗ all nodes, all scopes' `GroupState` in RAM | +| Rank-centrality | ✓ already scoped per parent; reads in-memory `GroupState` | + +**Measured RSS (release, ~395 imports, 2.8MB JSONL):** + +| Scenario | RSS | +|----------|-----| +| Empty data dir | ~14 MB (+ RocksDB baseline) | +| After boot with data | ~21–22 MB | +| Pre-offload (in-tree payloads + vec replay) | ~25–34 MB | + +Payload offload + streaming replay helped startup peak, but **the full in-memory reducer** is still the scaling ceiling. + +--- + +## What lives where (target) + +### JSONL (`events.jsonl`) + +All mutations, append-only: + +- `VoteRecorded` — scope, pair, ratios +- `EntityImported` — id, full upstream payload +- `NodeEnsured` — register path +- (legacy / other event types as present in log) + +### Durable / RocksDB (`{data_dir}/…`) + +Single embedded DB directory. Collections (names tentative): + +| Collection | Contents | Notes | +|------------|----------|-------| +| `entity_payloads` | `ItemId → JSON string` | **Done.** Fat Reddit API blobs | +| `nodes` | `ItemId → { data: EntityData, children: … }` | Small; no raw payload | +| `scopes/{parent}/…` | `GroupState` materialization | edges, voted_pairs, item_to_idx, recent_votes (capped) | + +Nested layout can follow durable's `Map → Map → Vec` patterns (see `durable/docs/001.md` Sorter sketch). + +### RAM + +| Resident | When | +|----------|------| +| Tokio, Axum, reqwest, RocksDB block cache (tuned) | always | +| **One scope slice** | per request (or LRU of few scopes, byte-capped) | +| Rank-centrality temporaries | during render for that scope | + +**Not** in RAM at steady state: all subreddits, all vote graphs, all payloads. + +--- + +## Write path + +Order matters: + +1. Append event to `events.jsonl` (must durably succeed first) +2. Apply event to durable projection (same logic as today's `apply_event`) +3. Invalidate / update in-memory scope cache if that scope is hot + +Journal worker already serializes votes disk → tree; extend to **disk → durable** instead of (eventually) **disk → full GlobalTree**. + +```rust +// conceptual +append(jsonl, event)?; +apply_to_durable(event)?; +scope_cache.invalidate(scope_for(event)); +``` + +On failure after (1): replay from log repairs projection on next boot or via `replay-index` command. + +--- + +## Read path + +For a page under parent scope `P` (e.g. `reddit.com/r/rust`): + +1. **Load scope** from durable (or scope LRU hit) + - `EntityData` + children for listing + - `GroupState` for ranking and pair selection +2. **Run rank-centrality** on that `GroupState` (requires RAM — that's fine) +3. **Render** +4. **Drop** scope from RAM or return to LRU + +Payload fetch (rare): `entity_store.get(id)` only when render needs fields not in `EntityData`. + +--- + +## Startup & recovery + +### Normal startup + +``` +open entity_db (RocksDB) +open / validate scope indexes in same DB +do NOT replay JSONL into RAM +serve requests (cold scopes loaded on demand) +``` + +### Rebuild projection + +``` +stream events.jsonl → apply_event → durable +(one line at a time; same as EventLog::replay today) +``` + +Run when: + +- First deploy of projection layer +- Detected corruption / missing durable dir +- Manual `replay-index` after restoring JSONL from backup + +JSONL is the only file you need to trust for recovery. + +--- + +## Scope cache (RAM bound) + +**Strict mode:** one scope in RAM at a time — simplest, lowest RAM. + +**Practical mode:** LRU cache with **byte budget** (e.g. 64–128MB for scopes on a 256MB VM): + +- Evict least-recently-used scope's `GroupState` + child views +- Reload from durable on next visit + +Eviction policy is independent of storage engine. + +--- + +## Rank-centrality + +No change to the algorithm. It already assumes a whole `GroupState` for one parent scope. + +Moving reducer to durable does **not** remove RC memory cost — it removes **holding every scope's graph at once**. + +Optional later: materialized score vectors on disk, invalidated on vote. Not required for v1 of this plan. + +--- + +## Durable mutations (future API) + +Separate **intent** from **apply** for batching and testability: + +```rust +let m = rankings.path().key("rust").key(day).push_end(score); +db.apply(m)?; // or batch.apply(&[m1, m2, m3]) +``` + +Benefits: + +- One RocksDB `WriteBatch` / one WAL flush per vote or import batch +- Serializable ops for tests +- Aligns with JSONL events at the app layer and storage ops at the durable layer + +Keep chained `entry().push()` as sugar over `path().…; apply()`. + +Type safety: use type-state path builders if we want compile-time nesting; erased `Vec` only if we accept runtime errors at apply. + +--- + +## Storage engine: RocksDB vs alternatives + +**Current choice:** RocksDB via vendored `durable/` workspace crate. + +| Engine | Verdict for sorter2 | +|--------|---------------------| +| **RocksDB** | Good default for LSM, prefix scans, write-heavy votes + bulk imports. C++ dep, tune block cache for 256MB. | +| **sled** | Pure Rust appeal; production reliability history gives pause. Not a priority switch. | +| **fjall** | Pure Rust LSM; evaluate with benchmarks if leaving RocksDB. | +| **redb** | Lighter pure Rust; fine for payload-only store; less ideal for heavy scattered writes across scopes. | +| **SQLite** | Wrong shape for nested fractal tree; OK for a single KV table only. | + +**Switching engines matters less than:** + +1. Batched durable writes +2. Not materializing full reducer in RAM +3. Scope-local load/evict + +RocksDB stays **narrow**: blob attic + materialized reducer projection. Not a replacement for JSONL. + +--- + +## Scaling story (before vs after) + +| | In-memory reducer (today) | Target (JSONL + durable projection) | +|--|---------------------------|-------------------------------------| +| **RAM grows with** | Total nodes + all scopes + (was) payloads | Hot scope count × scope size + runtime | +| **Disk grows with** | JSONL (+ entity_db payloads today) | JSONL + full durable projection | +| **Startup** | O(events) replay into RAM | O(1) open DB | +| **Fails when** | RSS > VM limit | Scope too large for one RC graph, or disk full | +| **Recovery** | Replay JSONL | Replay JSONL → rebuild durable | + +**Rough RAM per active subreddit (~500 posts, moderate votes):** + +| Component | Order of magnitude | +|-----------|-------------------| +| `EntityData` × 500 | 0.5–2 MB | +| `GroupState` | 1–5 MB | +| RC scratch | 1–5 MB | +| Runtime + tuned RocksDB | 15–25 MB | +| **Total one hot scope** | **~20–40 MB** | + +Multiple subreddits fit on 256MB with LRU eviction, not all resident at once. + +--- + +## Implementation phases + +### Phase 0 — Done + +- [x] Vend `durable` as workspace crate (`durable/`) +- [x] `entity_store`: payloads in `{data_dir}/entity_db` +- [x] Remove `entity_raw` from `NodeState` +- [x] `EventLog::replay`: stream JSONL, one `Event` at a time +- [x] `apply_event` in `state.rs` for replay semantics + +### Phase 1 — Durable projection (write path) + +- [ ] Single `Db` under `{data_dir}/store` (payloads + reducer) +- [ ] `apply_event` writes to durable collections (nodes, scopes) in addition to or instead of `GlobalTree` +- [ ] Journal + Reddit import paths use same apply +- [ ] Batched writes where possible (one flush per vote / per import batch) +- [ ] Tests: apply event → read back from durable + +### Phase 2 — Stop full tree at boot + +- [ ] Startup: open durable only; no `GlobalTree::new()` + full replay into RAM +- [ ] `replay-index` command / flag: stream JSONL → durable (offline rebuild) +- [ ] Integration tests use replay-index fixture or temp DB + +### Phase 3 — Scope load on read + +- [ ] `ScopeView { parent, children, group }` loaded from durable +- [ ] HTML / vote / pair / fetch handlers take `ScopeView` instead of `&GlobalTree` +- [ ] Remove or shrink `Arc>` + +### Phase 4 — Scope cache + +- [ ] LRU with byte budget for hot scopes +- [ ] Invalidate on write to that scope +- [ ] Metrics: cache hit/miss, evictions, scope load time + +### Phase 5 — Durable API polish (optional) + +- [ ] Reified path/mutation API + `WriteBatch` integration in `durable` +- [ ] RocksDB tuning preset for 256MB Fly (`block_cache`, write buffers) +- [ ] Document `replay-index` in AGENTS.md + +--- + +## Non-goals (for now) + +- Distributed replication or multi-writer +- Replacing JSONL as canonical store +- SQL query layer over state +- Materialized rank scores on disk (unless RC latency forces it) +- Switching from RocksDB to sled without benchmarks + +--- + +## Open questions + +1. **One DB or two?** `{data_dir}/entity_db` today vs single `{data_dir}/store` — merge on Phase 1? +2. **Scope key encoding** — string `ItemId` paths vs hashed; must match event `scope` field. +3. **Child scope wiring** — Reddit `apply_entity_under_parent` creates children without full path ensure; durable schema must preserve this. +4. **Clojure smoke tests** — still read `events.jsonl`; durable is internal. No change expected. +5. **Fly volume** — `/data` holds JSONL + RocksDB; monitor disk alongside RAM. + +--- + +## Summary + +**JSONL = write log. Durable = full reducer on disk + fat payloads. RAM = one rank-centrality graph (or a small LRU of scopes).** + +Option B from design discussions is not separate from “reducer in durable” — it **is** reducer in durable, with JSONL still owning writes and recovery. The incremental work is: projection on apply, drop full tree at boot, scope load on read, then cache with eviction. Side B — contributor: tommy-mor Side B — commit message: [a45842df] better dev iteration script Side B — unified diff (full patch): diff --git a/.gitignore b/.gitignore index 9cd2eaece834481bee87639cd7cb6be77f33e302..b70cf5b97cd3556e061233d8b0aefd333568a991 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,7 @@ target/ repomix-output.xml psalms_kjv.txt dev-data/ +fixture-data/ .worktrees/ .specstory/ .specstory/ diff --git a/bb.edn b/bb.edn index 5d1d642d75d03f70879d7673e7ad95360336a8b1..4680e82d4227057d2eea6041dc5647beb759131d 100644 --- a/bb.edn +++ b/bb.edn @@ -47,7 +47,7 @@ "RUST_LOG" "info"})})))} fixture - {:doc "Run local server + mock OAuth + seeded walkthrough data for manual browser demos" + {:doc "Run local server via cargo-watch + mock OAuth + seeded walkthrough data (persistent ./fixture-data/, prefers PORT 8080). Requires cargo-watch." :requires ([test.walkthrough-fixture :as walkthrough-fixture]) :task (walkthrough-fixture/run-fixture)} diff --git a/server/src/html/forum.rs b/server/src/html/forum.rs index e00e7110e2ad778d63b3d00b0cc4ffe20ba2e0d4..815fc6529062b103553644fc15e516334aec2b88 100644 --- a/server/src/html/forum.rs +++ b/server/src/html/forum.rs @@ -976,12 +976,6 @@ pub async fn room_page( html! { (strip) nav class="breadcrumb" { (bc_room(&nav, slug_display, None)) } - h2 { (slug_display) } - p class="muted" { (room_id) } - p class="muted room-links" { - "room garden · " - a href=(nav.garden_root_url()) { "~" } - } (room_members_section_markup(&reduced, &room_id, false)) h3 { "threads" } (render_thread_feed(Some(&nav), "room-thread-feed", &rows, now)) diff --git a/test/common.clj b/test/common.clj index 76a3e99fdb67c10bcb5c2f53d8bcd0917398af50..c6deba0d90995b0e3f9c4b8ad66a516ed19b870e 100644 --- a/test/common.clj +++ b/test/common.clj @@ -107,6 +107,16 @@ (.close ss) port)) +(defn pick-port-prefer + "Use `preferred` if it can be bound, otherwise an ephemeral port (same as `pick-port`)." + [preferred] + (try + (let [ss (java.net.ServerSocket. preferred)] + (.close ss) + preferred) + (catch java.io.IOException _ + (pick-port)))) + (defn wait-for-server "Poll /healthz until it returns 'ok', up to `timeout-ms`." [base-url timeout-ms] diff --git a/test/walkthrough_fixture.clj b/test/walkthrough_fixture.clj index 4b814822029c913e38151ca53848d1c64fb58373..a487ff1ef8548580577914fb33f321586859c9af 100644 --- a/test/walkthrough_fixture.clj +++ b/test/walkthrough_fixture.clj @@ -1,11 +1,18 @@ (ns test.walkthrough-fixture "Launch a local slug server with mock OAuth and seed browser-friendly demo data." (:require [babashka.fs :as fs] + [babashka.process :as p] [cheshire.core :as json] [clojure.string :as str] [test.common :as common] [test.oauth :as oauth])) +(def ^:private fixture-data-dir "fixture-data") +;; Prefer the same port as `bb dev` / `bb watch`; fall back if something else is listening. +(def ^:private preferred-slug-port 8080) +;; First `cargo watch` compile can exceed a release-binary startup; allow several minutes. +(def ^:private health-wait-ms 300000) + (defn- assert! [pred msg] (when-not pred (throw (ex-info msg {})))) @@ -85,49 +92,87 @@ :thread_url (str base-url "/r/" room-short "/" room-slug "/t/walkthrough-thread") :garden_url (str base-url "/r/" room-short "/" room-slug "/~/secret/item")}})) +(defn- rebase-fixture-summary [saved current-base-url current-google-url current-data-dir] + (let [inner (:summary saved) + room (:room inner) + rs (:short room) + lg (:slug room)] + (assoc saved + :base_url current-base-url + :mock_google_url current-google-url + :data_dir (str current-data-dir) + :summary (assoc inner + :room (assoc room + :url (str current-base-url "/r/" rs "/" lg) + :thread_url (str current-base-url "/r/" rs "/" lg "/t/walkthrough-thread") + :garden_url (str current-base-url "/r/" rs "/" lg "/~/secret/item")))))) + +(defn- fixture-log-present? [data-dir] + (let [p (fs/path data-dir "events.jsonl")] + (and (fs/exists? p) (pos? (fs/size p))))) + +(defn- load-or-seed-summary! + [base-url google-url data-dir summary-path] + (if (and (fixture-log-present? data-dir) (fs/exists? summary-path)) + (let [saved (json/parse-string (slurp summary-path) true) + rebased (rebase-fixture-summary saved base-url google-url data-dir)] + (spit summary-path (json/generate-string rebased {:pretty true})) + (println "") + (println "reusing fixture-data/ (delete the directory for a fresh seed)") + rebased) + (let [seeded (seed-demo! base-url) + s {:base_url base-url + :mock_google_url google-url + :data_dir (str data-dir) + :summary seeded}] + (spit summary-path (json/generate-string s {:pretty true})) + s))) + (defn run-fixture [& _args] - (let [build (common/run-cargo-build-release! ["slugsocial-server"]) - _ (assert! (zero? (:exit build)) "cargo build --release failed") - server-bin "target/release/slugsocial-server" - tmp-dir (str (fs/create-temp-dir {:prefix "slug-walkthrough-"})) - slug-port (common/pick-port) + (let [data-dir (str (fs/absolutize (fs/path (fs/cwd) fixture-data-dir))) + slug-port (common/pick-port-prefer preferred-slug-port) google-port (common/pick-port) base-url (str "http://127.0.0.1:" slug-port) google-url (str "http://127.0.0.1:" google-port) - stable-dir "/tmp/slug-walkthrough-fixture" - summary-path (str stable-dir "/summary.json") + summary-path (str (fs/path data-dir "summary.json")) !server (atom nil) !google (atom nil) - server-env (common/slug-server-env tmp-dir base-url google-url slug-port)] + server-env (merge (common/slug-server-env data-dir base-url google-url slug-port) + {"RUST_LOG" "info"}) + watch-cmd [(common/cargo-bin) "watch" + "-x" "run -p slugsocial-server" + "-w" "server/src" + "-w" "server/static" + "-w" "types/src"]] (try - (fs/create-dirs stable-dir) + (fs/create-dirs data-dir) (reset! !google (oauth/start-mock-google google-port :google-users ["google-user-alice" "google-user-bob"])) - (reset! !server (common/start-server server-bin server-env)) - (assert! (common/wait-for-server base-url 10000) "server did not become healthy") - (let [seeded (seed-demo! base-url) - summary {:base_url base-url - :mock_google_url google-url - :data_dir tmp-dir - :summary seeded}] - (spit summary-path (json/generate-string summary {:pretty true})) + (println "") + (println "starting cargo-watch (first compile may take a while)…") + (flush) + (reset! !server (p/process watch-cmd {:inherit true :env server-env})) + (assert! (common/wait-for-server base-url health-wait-ms) "server did not become healthy") + (let [summary (load-or-seed-summary! base-url google-url data-dir summary-path)] (println "") (println "walkthrough fixture ready") (println (str " base url: " base-url)) (println (str " room page: " (get-in summary [:summary :room :url]))) (println (str " thread page: " (get-in summary [:summary :room :thread_url]))) (println (str " garden page: " (get-in summary [:summary :room :garden_url]))) + (println (str " data dir: " data-dir)) (println (str " summary json: " summary-path)) (println "") (println "seeded users") (println " alice / bob via mock OAuth") (println "") + (println "editing server/src or server/static reloads the server; data persists in fixture-data/") + (println "") (println "press Ctrl-C to stop") (flush) (while true (Thread/sleep 1000))) (finally (when-some [s @!server] (common/kill-server s)) - (when-some [g @!google] ((:stop-fn g))) - (fs/delete-tree tmp-dir))))) + (when-some [g @!google] ((:stop-fn g)))))))