{"messages":[{"content":"You are a constitutional council ranking individual git commits for ownership allocation.\n\nCompare these two commits. Decide which contributed more lasting value to the project.\n\nJudge substance, not spectacle:\n- Prefer correct, lasting design and real bugfixes over churn, formatting, renames, or generated noise.\n- Prefer clarity and necessity over sheer line count. A small precise change can beat a large diffuse one.\n- Do not favor a side merely because its patch is longer or noisier.\n- Weight what the change does for the project, not the contributor's name.\n\nReturn ONLY a JSON object: {\"winner\": \"A\" or \"B\", \"ratio\": \"N:M\", \"explanation\": \"...\"}\nThe explanation must cite concrete differences in the patches (1-3 sentences).\n\nSide A — contributor: tommy-mor\nSide A — commit message:\n[432e1450] nice\n\nSide A — unified diff (full patch):\ndiff --git a/reddit.rs b/reddit.rs\nnew file mode 100644\nindex 0000000000000000000000000000000000000000..da27f6f76482dd27d44cad9d7b844972bf042e64\n--- /dev/null\n+++ b/reddit.rs\n@@ -0,0 +1,420 @@\n+use governor::{Quota, RateLimiter, Jitter};\n+use nonzero_ext::nonzero;\n+use reqwest::{Client, StatusCode};\n+use serde::{Deserialize, Serialize};\n+use std::sync::Arc;\n+use std::time::Duration;\n+use anyhow::{Result, Context, anyhow};\n+\n+/// Reddit API client with built-in rate limiting\n+pub struct RedditClient {\n+    client: Client,\n+    limiter: Arc<governor::DefaultDirectRateLimiter>,\n+    user_agent: String,\n+}\n+\n+impl RedditClient {\n+    /// Create a new Reddit client with rate limiting\n+    /// Reddit API allows 60 requests per minute for OAuth2 authenticated apps\n+    /// We'll be conservative and use 50 requests per minute\n+    pub fn new() -> Self {\n+        // Create rate limiter: 50 requests per minute\n+        let quota = Quota::per_minute(nonzero!(50u32));\n+        let limiter = Arc::new(RateLimiter::direct(quota));\n+        \n+        // Create HTTP client with timeout\n+        let client = Client::builder()\n+            .timeout(Duration::from_secs(30))\n+            .build()\n+            .expect(\"Failed to create HTTP client\");\n+        \n+        // Reddit requires a unique user agent\n+        let user_agent = format!(\n+            \"rust:sorter:v{} (by /u/hourLong_arnould)\",\n+            env!(\"CARGO_PKG_VERSION\")\n+        );\n+        \n+        Self {\n+            client,\n+            limiter,\n+            user_agent,\n+        }\n+    }\n+    \n+    /// Fetch hot posts from a subreddit\n+    pub async fn get_subreddit_hot(&self, subreddit: &str, limit: usize) -> Result<RedditListingResponse> {\n+        // Wait for rate limiter\n+        self.limiter.until_ready_with_jitter(Jitter::up_to(Duration::from_millis(100))).await;\n+        \n+        let url = format!(\"https://www.reddit.com/r/{}/hot.json?limit={}\", subreddit, limit);\n+        \n+        let response = self.client\n+            .get(&url)\n+            .header(\"User-Agent\", &self.user_agent)\n+            .send()\n+            .await\n+            .context(\"Failed to send request to Reddit\")?;\n+        \n+        self.handle_response(response).await\n+    }\n+    \n+    /// Fetch top posts from a subreddit\n+    pub async fn get_subreddit_top(&self, subreddit: &str, limit: usize, time_period: &str) -> Result<RedditListingResponse> {\n+        self.limiter.until_ready_with_jitter(Jitter::up_to(Duration::from_millis(100))).await;\n+        \n+        let url = format!(\n+            \"https://www.reddit.com/r/{}/top.json?limit={}&t={}\",\n+            subreddit, limit, time_period\n+        );\n+        \n+        let response = self.client\n+            .get(&url)\n+            .header(\"User-Agent\", &self.user_agent)\n+            .send()\n+            .await\n+            .context(\"Failed to send request to Reddit\")?;\n+        \n+        self.handle_response(response).await\n+    }\n+    \n+    /// Fetch new posts from a subreddit\n+    pub async fn get_subreddit_new(&self, subreddit: &str, limit: usize) -> Result<RedditListingResponse> {\n+        self.limiter.until_ready_with_jitter(Jitter::up_to(Duration::from_millis(100))).await;\n+        \n+        let url = format!(\"https://www.reddit.com/r/{}/new.json?limit={}\", subreddit, limit);\n+        \n+        let response = self.client\n+            .get(&url)\n+            .header(\"User-Agent\", &self.user_agent)\n+            .send()\n+            .await\n+            .context(\"Failed to send request to Reddit\")?;\n+        \n+        self.handle_response(response).await\n+    }\n+    \n+    /// Fetch a specific post and its comments\n+    pub async fn get_post(&self, subreddit: &str, post_id: &str) -> Result<Vec<RedditListingResponse>> {\n+        self.limiter.until_ready_with_jitter(Jitter::up_to(Duration::from_millis(100))).await;\n+        \n+        let url = format!(\n+            \"https://www.reddit.com/r/{}/comments/{}.json\",\n+            subreddit, post_id\n+        );\n+        \n+        let response = self.client\n+            .get(&url)\n+            .header(\"User-Agent\", &self.user_agent)\n+            .send()\n+            .await\n+            .context(\"Failed to send request to Reddit\")?;\n+        \n+        match response.status() {\n+            StatusCode::OK => {\n+                let listings: Vec<RedditListingResponse> = response\n+                    .json()\n+                    .await\n+                    .context(\"Failed to parse Reddit response\")?;\n+                Ok(listings)\n+            }\n+            StatusCode::TOO_MANY_REQUESTS => {\n+                Err(anyhow!(\"Reddit rate limit exceeded. Please try again later.\"))\n+            }\n+            StatusCode::NOT_FOUND => {\n+                Err(anyhow!(\"Post not found: r/{}/comments/{}\", subreddit, post_id))\n+            }\n+            status => {\n+                Err(anyhow!(\"Reddit API error: {}\", status))\n+            }\n+        }\n+    }\n+    \n+    /// Fetch user profile (posts and comments)\n+    pub async fn get_user_profile(&self, username: &str, content_type: &str, limit: usize) -> Result<RedditListingResponse> {\n+        self.limiter.until_ready_with_jitter(Jitter::up_to(Duration::from_millis(100))).await;\n+        \n+        let url = format!(\n+            \"https://www.reddit.com/user/{}/{}.json?limit={}\",\n+            username, content_type, limit\n+        );\n+        \n+        let response = self.client\n+            .get(&url)\n+            .header(\"User-Agent\", &self.user_agent)\n+            .send()\n+            .await\n+            .context(\"Failed to send request to Reddit\")?;\n+        \n+        self.handle_response(response).await\n+    }\n+    \n+    /// Handle Reddit API response with proper error checking\n+    async fn handle_response(&self, response: reqwest::Response) -> Result<RedditListingResponse> {\n+        match response.status() {\n+            StatusCode::OK => {\n+                let listing: RedditListingResponse = response\n+                    .json()\n+                    .await\n+                    .context(\"Failed to parse Reddit response\")?;\n+                Ok(listing)\n+            }\n+            StatusCode::TOO_MANY_REQUESTS => {\n+                Err(anyhow!(\"Reddit rate limit exceeded. Please try again later.\"))\n+            }\n+            StatusCode::NOT_FOUND => {\n+                Err(anyhow!(\"Reddit resource not found\"))\n+            }\n+            StatusCode::FORBIDDEN => {\n+                Err(anyhow!(\"Access forbidden. The subreddit may be private.\"))\n+            }\n+            status => {\n+                Err(anyhow!(\"Reddit API error: {}\", status))\n+            }\n+        }\n+    }\n+}\n+\n+// Reddit API response types\n+#[derive(Debug, Clone, Serialize, Deserialize)]\n+pub struct RedditListingResponse {\n+    pub kind: String,\n+    pub data: RedditListingData,\n+}\n+\n+#[derive(Debug, Clone, Serialize, Deserialize)]\n+pub struct RedditListingData {\n+    #[serde(skip_serializing_if = \"Option::is_none\")]\n+    pub after: Option<String>,\n+    #[serde(skip_serializing_if = \"Option::is_none\")]\n+    pub before: Option<String>,\n+    pub children: Vec<RedditThing>,\n+    #[serde(skip_serializing_if = \"Option::is_none\")]\n+    pub modhash: Option<String>,\n+}\n+\n+#[derive(Debug, Clone, Serialize, Deserialize)]\n+pub struct RedditThing {\n+    pub kind: String,\n+    pub data: RedditThingData,\n+}\n+\n+/// Reddit's \"Thing\" data can be either a post, comment, or other types\n+/// We use untagged enum to handle different data structures\n+#[derive(Debug, Clone, Serialize, Deserialize)]\n+#[serde(untagged)]\n+pub enum RedditThingData {\n+    Post(RedditPost),\n+    Comment(RedditComment),\n+    // For things we don't care about yet (like \"more\" comments)\n+    Other(serde_json::Value),\n+}\n+\n+/// Reddit post data with serde handling all the parsing\n+#[derive(Debug, Clone, Serialize, Deserialize)]\n+pub struct RedditPost {\n+    pub id: String,\n+    pub name: String, // Full name (t3_xxx)\n+    #[serde(default = \"default_untitled\")]\n+    pub title: String,\n+    #[serde(default = \"default_deleted_user\")]\n+    pub author: String,\n+    pub subreddit: String,\n+    #[serde(default)]\n+    pub url: String,\n+    #[serde(default)]\n+    pub permalink: String,\n+    #[serde(default, deserialize_with = \"deserialize_selftext\")]\n+    pub selftext: Option<String>,\n+    #[serde(default)]\n+    pub score: i64,\n+    #[serde(default)]\n+    pub num_comments: i64,\n+    #[serde(default)]\n+    pub created_utc: f64,\n+    #[serde(default, deserialize_with = \"deserialize_thumbnail\")]\n+    pub thumbnail: Option<String>,\n+    #[serde(default)]\n+    pub is_video: bool,\n+    #[serde(default)]\n+    pub is_self: bool,\n+    // Additional useful fields\n+    #[serde(default)]\n+    pub ups: i64,\n+    #[serde(default)]\n+    pub downs: i64,\n+    #[serde(default)]\n+    pub upvote_ratio: f64,\n+    #[serde(default)]\n+    pub over_18: bool,\n+    #[serde(default)]\n+    pub spoiler: bool,\n+    #[serde(default)]\n+    pub stickied: bool,\n+    #[serde(default)]\n+    pub locked: bool,\n+    #[serde(default)]\n+    pub distinguished: Option<String>,\n+}\n+\n+/// Reddit comment data\n+#[derive(Debug, Clone, Serialize, Deserialize)]\n+pub struct RedditComment {\n+    pub id: String,\n+    pub name: String, // Full name (t1_xxx)\n+    #[serde(default = \"default_deleted_user\")]\n+    pub author: String,\n+    #[serde(default)]\n+    pub body: String,\n+    #[serde(default)]\n+    pub score: i64,\n+    #[serde(default)]\n+    pub created_utc: f64,\n+    #[serde(default)]\n+    pub parent_id: String,\n+    #[serde(default)]\n+    pub permalink: String,\n+    #[serde(default)]\n+    pub depth: i32,\n+    // Additional useful fields\n+    #[serde(default)]\n+    pub ups: i64,\n+    #[serde(default)]\n+    pub downs: i64,\n+    #[serde(default)]\n+    pub edited: RedditEditStatus,\n+    #[serde(default)]\n+    pub stickied: bool,\n+    #[serde(default)]\n+    pub distinguished: Option<String>,\n+    #[serde(default)]\n+    pub is_submitter: bool,\n+    #[serde(default)]\n+    pub collapsed: bool,\n+    #[serde(default)]\n+    pub controversiality: i32,\n+}\n+\n+/// Reddit's edit status - can be false or a timestamp\n+#[derive(Debug, Clone, Serialize, Deserialize)]\n+#[serde(untagged)]\n+pub enum RedditEditStatus {\n+    NotEdited(bool),\n+    EditedAt(f64),\n+}\n+\n+impl Default for RedditEditStatus {\n+    fn default() -> Self {\n+        RedditEditStatus::NotEdited(false)\n+    }\n+}\n+\n+// Helper functions for serde defaults\n+fn default_untitled() -> String {\n+    \"Untitled\".to_string()\n+}\n+\n+fn default_deleted_user() -> String {\n+    \"[deleted]\".to_string()\n+}\n+\n+// Custom deserializer for selftext (empty strings should be None)\n+fn deserialize_selftext<'de, D>(deserializer: D) -> Result<Option<String>, D::Error>\n+where\n+    D: serde::Deserializer<'de>,\n+{\n+    let s: Option<String> = Option::deserialize(deserializer)?;\n+    Ok(s.filter(|text| !text.is_empty()))\n+}\n+\n+// Custom deserializer for thumbnail (filter out \"self\", \"default\", empty)\n+fn deserialize_thumbnail<'de, D>(deserializer: D) -> Result<Option<String>, D::Error>\n+where\n+    D: serde::Deserializer<'de>,\n+{\n+    let s: Option<String> = Option::deserialize(deserializer)?;\n+    Ok(s.filter(|thumb| {\n+        !thumb.is_empty() && thumb != \"self\" && thumb != \"default\" && thumb != \"nsfw\" && thumb != \"spoiler\"\n+    }))\n+}\n+\n+// Helper methods for extracting typed data from Things\n+impl RedditThing {\n+    /// Try to get this Thing as a Post\n+    pub fn as_post(&self) -> Option<&RedditPost> {\n+        if self.kind != \"t3\" {\n+            return None;\n+        }\n+        match &self.data {\n+            RedditThingData::Post(post) => Some(post),\n+            _ => None,\n+        }\n+    }\n+    \n+    /// Try to get this Thing as a Comment\n+    pub fn as_comment(&self) -> Option<&RedditComment> {\n+        if self.kind != \"t1\" {\n+            return None;\n+        }\n+        match &self.data {\n+            RedditThingData::Comment(comment) => Some(comment),\n+            _ => None,\n+        }\n+    }\n+    \n+    /// Check if this is a \"more comments\" indicator\n+    pub fn is_more_comments(&self) -> bool {\n+        self.kind == \"more\"\n+    }\n+}\n+\n+// Helper methods for working with listing responses\n+impl RedditListingResponse {\n+    /// Extract all posts from this listing\n+    pub fn get_posts(&self) -> Vec<&RedditPost> {\n+        self.data.children.iter()\n+            .filter_map(|thing| thing.as_post())\n+            .collect()\n+    }\n+    \n+    /// Extract all comments from this listing\n+    pub fn get_comments(&self) -> Vec<&RedditComment> {\n+        self.data.children.iter()\n+            .filter_map(|thing| thing.as_comment())\n+            .collect()\n+    }\n+    \n+    /// Extract posts as owned values\n+    pub fn into_posts(self) -> Vec<RedditPost> {\n+        self.data.children.into_iter()\n+            .filter_map(|thing| {\n+                if thing.kind == \"t3\" {\n+                    match thing.data {\n+                        RedditThingData::Post(post) => Some(post),\n+                        _ => None,\n+                    }\n+                } else {\n+                    None\n+                }\n+            })\n+            .collect()\n+    }\n+    \n+    /// Extract comments as owned values\n+    pub fn into_comments(self) -> Vec<RedditComment> {\n+        self.data.children.into_iter()\n+            .filter_map(|thing| {\n+                if thing.kind == \"t1\" {\n+                    match thing.data {\n+                        RedditThingData::Comment(comment) => Some(comment),\n+                        _ => None,\n+                    }\n+                } else {\n+                    None\n+                }\n+            })\n+            .collect()\n+    }\n+}\n+\n+\n+\n\n\nSide B — contributor: tommy-mor\nSide B — commit message:\n[461bfaa7] finished plan\n\nSide B — unified diff (full patch):\ndiff --git a/PLAN.md b/PLAN.md\ndeleted file mode 100644\nindex bc7b3beecea4d63d723432b479a003967f910e45..0000000000000000000000000000000000000000\n--- a/PLAN.md\n+++ /dev/null\n@@ -1,363 +0,0 @@\n-# sorter2 storage & memory plan\n-\n-## Goal\n-\n-Fit a **decent chunk of Reddit** into a **256MB** Fly VM while keeping the product simple: one Rust binary, no external database service.\n-\n-**RAM should be bounded by query shape**, not dataset size — ideally one rank-centrality graph in memory at a time, plus runtime overhead.\n-\n-**`events.jsonl` remains the main database.** Everything on disk elsewhere is a **rebuildable projection**.\n-\n----\n-\n-## Architecture (target)\n-\n-```\n-                    ┌─────────────────────────────────┐\n-                    │  events.jsonl (source of truth) │\n-                    └───────────────┬─────────────────┘\n-                                    │\n-                         append on every mutation\n-                                    │\n-                                    ▼\n-         ┌──────────────────────────────────────────────┐\n-         │  apply event → durable projection (on disk)     │\n-         │  (same semantics as today's in-memory reducer)  │\n-         └──────────────────────────────────────────────┘\n-                    │                    │\n-                    ▼                    ▼\n-         ┌──────────────────┐   ┌──────────────────────────┐\n-         │ entity_payloads  │   │ reducer state per scope   │\n-         │ (fat Reddit JSON)│   │ nodes, children, edges, … │\n-         └──────────────────┘   └──────────────────────────┘\n-                    │                    │\n-                    └────────┬───────────┘\n-                             ▼\n-         ┌──────────────────────────────────────────────┐\n-         │  RAM per request (or small LRU cache)         │\n-         │  • one scope's GroupState for rank-centrality │\n-         │  • children + EntityData (small views)        │\n-         │  • RC scratch allocations                     │\n-         │  → compute → render → drop / evict            │\n-         └──────────────────────────────────────────────┘\n-```\n-\n-This is **event sourcing / CQRS**:\n-\n-| Layer | Role |\n-|-------|------|\n-| **JSONL** | Canonical write log; audit; disaster recovery |\n-| **Durable (RocksDB)** | Materialized read model + payload store; rebuildable from JSONL |\n-| **RAM** | One (or few) hot scopes for ranking and render |\n-\n-Durable is **not** a second source of truth. If projection and log diverge, **stream JSONL and rebuild durable**.\n-\n-The plan is sound, but only if the projection layer is treated as a crash-recoverable index:\n-\n-- JSONL append must mean \"the bytes are recoverable after process or VM crash\" (`flush` alone is not enough; use `sync_data` / fsync-equivalent on the append path or make an explicit weaker durability tradeoff).\n-- Durable projection writes are allowed to lag JSONL, but startup must catch up from the last applied event before serving reads.\n-- Event application must be exactly-once for the durable projection. Votes add edge weights, so accidentally replaying the same event twice changes rankings.\n-- Projection schema should stay boring and explicit: persist nodes, children, edge weights, voted pairs, and capped recent votes directly. Avoid clever nested abstractions if they make rebuilds, migrations, or audits harder.\n-\n----\n-\n-## What we have today (baseline)\n-\n-| Piece | Status |\n-|-------|--------|\n-| `events.jsonl` append-only log | ✓ source of truth |\n-| `EventLog::replay` streaming one event at a time | ✓ no full `Vec<Event>` at startup |\n-| `entity_store` / `entity_db` (RocksDB via `durable`) | ✓ fat payloads off-heap |\n-| `EntityData` in `GlobalTree` | ✓ small derived views in RAM |\n-| Full `GlobalTree` replayed at boot | ✗ all nodes, all scopes' `GroupState` in RAM |\n-| Rank-centrality | ✓ already scoped per parent; reads in-memory `GroupState` |\n-\n-**Measured RSS (release, ~395 imports, 2.8MB JSONL):**\n-\n-| Scenario | RSS |\n-|----------|-----|\n-| Empty data dir | ~14 MB (+ RocksDB baseline) |\n-| After boot with data | ~21–22 MB |\n-| Pre-offload (in-tree payloads + vec replay) | ~25–34 MB |\n-\n-Payload offload + streaming replay helped startup peak, but **the full in-memory reducer** is still the scaling ceiling.\n-\n----\n-\n-## What lives where (target)\n-\n-### JSONL (`events.jsonl`)\n-\n-All mutations, append-only:\n-\n-- `VoteRecorded` — scope, pair, ratios\n-- `EntityImported` — id, full upstream payload\n-- `NodeEnsured` — register path\n-- (legacy / other event types as present in log)\n-\n-### Durable / RocksDB (`{data_dir}/…`)\n-\n-Single embedded DB directory. Collections (names tentative):\n-\n-| Collection | Contents | Notes |\n-|------------|----------|-------|\n-| `entity_payloads` | `ItemId → JSON string` | **Done.** Fat Reddit API blobs |\n-| `nodes` | `ItemId → { data: EntityData, children: … }` | Small; no raw payload |\n-| `scopes/{parent}/…` | `GroupState` materialization | edges, voted_pairs, item_to_idx, recent_votes (capped) |\n-\n-Nested layout can follow durable's `Map → Map → Vec` patterns (see `durable/docs/001.md` Sorter sketch).\n-\n-### RAM\n-\n-| Resident | When |\n-|----------|------|\n-| Tokio, Axum, reqwest, RocksDB block cache (tuned) | always |\n-| **One scope slice** | per request (or LRU of few scopes, byte-capped) |\n-| Rank-centrality temporaries | during render for that scope |\n-\n-**Not** in RAM at steady state: all subreddits, all vote graphs, all payloads.\n-\n----\n-\n-## Write path\n-\n-Order matters:\n-\n-1. Append event to `events.jsonl` (must durably succeed first)\n-2. Apply event to durable projection (same logic as today's `apply_event`)\n-3. Invalidate / update in-memory scope cache if that scope is hot\n-\n-Journal worker already serializes votes disk → tree; extend to **disk → durable** instead of (eventually) **disk → full GlobalTree**.\n-\n-```rust\n-// conceptual\n-append(jsonl, event)?;\n-apply_to_durable(event)?;\n-scope_cache.invalidate(scope_for(event));\n-```\n-\n-On failure after (1): replay from log repairs projection on next boot or via `replay-index` command.\n-\n-Because JSONL and RocksDB cannot be committed atomically together, durable must record a projection cursor alongside the projection:\n-\n-- Prefer a monotonically increasing event sequence number in each JSONL event.\n-- Acceptable first version: byte offset + line checksum, as long as truncation and partial trailing lines are handled deliberately.\n-- Update projection data and cursor in the same RocksDB `WriteBatch`.\n-- On startup, read the cursor, scan only the JSONL tail after that cursor, apply missing events, then serve.\n-- If the cursor is missing, corrupt, or points past the log, rebuild durable from JSONL.\n-\n-This keeps the append-first rule simple: if the process dies after JSONL append but before RocksDB apply, catch-up repairs it; if it dies after RocksDB apply but before cursor update, the batch should not expose a cursor that skips work.\n-\n----\n-\n-## Read path\n-\n-For a page under parent scope `P` (e.g. `reddit.com/r/rust`):\n-\n-1. **Load scope** from durable (or scope LRU hit)\n-   - `EntityData` + children for listing\n-   - `GroupState` for ranking and pair selection\n-2. **Run rank-centrality** on that `GroupState` (requires RAM — that's fine)\n-3. **Render**\n-4. **Drop** scope from RAM or return to LRU\n-\n-Payload fetch (rare): `entity_store.get(id)` only when render needs fields not in `EntityData`.\n-\n----\n-\n-## Startup & recovery\n-\n-### Normal startup\n-\n-```\n-open entity_db (RocksDB)\n-open / validate scope indexes in same DB\n-read projection cursor\n-stream only unapplied JSONL tail into durable\n-do NOT replay JSONL into RAM\n-serve requests (cold scopes loaded on demand)\n-```\n-\n-### Rebuild projection\n-\n-```\n-stream events.jsonl → apply_event → durable\n-(one line at a time; same as EventLog::replay today)\n-```\n-\n-Run when:\n-\n-- First deploy of projection layer\n-- Detected corruption / missing durable dir\n-- Manual `replay-index` after restoring JSONL from backup\n-\n-JSONL is the only file you need to trust for recovery.\n-\n----\n-\n-## Scope cache (RAM bound)\n-\n-**Strict mode:** one scope in RAM at a time — simplest, lowest RAM.\n-\n-**Practical mode:** LRU cache with **byte budget** (e.g. 64–128MB for scopes on a 256MB VM):\n-\n-- Evict least-recently-used scope's `GroupState` + child views\n-- Reload from durable on next visit\n-\n-Eviction policy is independent of storage engine.\n-\n----\n-\n-## Rank-centrality\n-\n-No change to the algorithm. It already assumes a whole `GroupState` for one parent scope.\n-\n-Moving reducer to durable does **not** remove RC memory cost — it removes **holding every scope's graph at once**.\n-\n-Optional later: materialized score vectors on disk, invalidated on vote. Not required for v1 of this plan.\n-\n----\n-\n-## Durable mutations (future API)\n-\n-Separate **intent** from **apply** for batching and testability:\n-\n-```rust\n-let m = rankings.path().key(\"rust\").key(day).push_end(score);\n-db.apply(m)?;  // or batch.apply(&[m1, m2, m3])\n-```\n-\n-Benefits:\n-\n-- One RocksDB `WriteBatch` / one WAL flush per vote or import batch\n-- Serializable ops for tests\n-- Aligns with JSONL events at the app layer and storage ops at the durable layer\n-\n-Keep chained `entry().push()` as sugar over `path().…; apply()`.\n-\n-Type safety: use type-state path builders if we want compile-time nesting; erased `Vec<Op>` only if we accept runtime errors at apply.\n-\n----\n-\n-## Storage engine: RocksDB vs alternatives\n-\n-**Current choice:** RocksDB via vendored `durable/` workspace crate.\n-\n-| Engine | Verdict for sorter2 |\n-|--------|---------------------|\n-| **RocksDB** | Good default for LSM, prefix scans, write-heavy votes + bulk imports. C++ dep, tune block cache for 256MB. |\n-| **sled** | Pure Rust appeal; production reliability history gives pause. Not a priority switch. |\n-| **fjall** | Pure Rust LSM; evaluate with benchmarks if leaving RocksDB. |\n-| **redb** | Lighter pure Rust; fine for payload-only store; less ideal for heavy scattered writes across scopes. |\n-| **SQLite** | Wrong shape for nested fractal tree; OK for a single KV table only. |\n-\n-**Switching engines matters less than:**\n-\n-1. Batched durable writes\n-2. Not materializing full reducer in RAM\n-3. Scope-local load/evict\n-\n-RocksDB stays **narrow**: blob attic + materialized reducer projection. Not a replacement for JSONL.\n-\n----\n-\n-## Scaling story (before vs after)\n-\n-| | In-memory reducer (today) | Target (JSONL + durable projection) |\n-|--|---------------------------|-------------------------------------|\n-| **RAM grows with** | Total nodes + all scopes + (was) payloads | Hot scope count × scope size + runtime |\n-| **Disk grows with** | JSONL (+ entity_db payloads today) | JSONL + full durable projection |\n-| **Startup** | O(events) replay into RAM | O(1) open DB |\n-| **Fails when** | RSS > VM limit | Scope too large for one RC graph, or disk full |\n-| **Recovery** | Replay JSONL | Replay JSONL → rebuild durable |\n-\n-**Rough RAM per active subreddit (~500 posts, moderate votes):**\n-\n-| Component | Order of magnitude |\n-|-----------|-------------------|\n-| `EntityData` × 500 | 0.5–2 MB |\n-| `GroupState` | 1–5 MB |\n-| RC scratch | 1–5 MB |\n-| Runtime + tuned RocksDB | 15–25 MB |\n-| **Total one hot scope** | **~20–40 MB** |\n-\n-Multiple subreddits fit on 256MB with LRU eviction, not all resident at once.\n-\n----\n-\n-## Implementation phases\n-\n-### Phase 0 — Done\n-\n-- [x] Vend `durable` as workspace crate (`durable/`)\n-- [x] `entity_store`: payloads in `{data_dir}/entity_db`\n-- [x] Remove `entity_raw` from `NodeState`\n-- [x] `EventLog::replay`: stream JSONL, one `Event` at a time\n-- [x] `apply_event` in `state.rs` for replay semantics\n-\n-### Phase 1 — Durable projection (write path)\n-\n-- [ ] Single `Db` under `{data_dir}/store` (payloads + reducer)\n-- [ ] JSONL append path uses explicit durable sync semantics (`sync_data` / fsync-equivalent) or documents any weaker mode\n-- [ ] Add projection cursor metadata (event sequence, or byte offset + checksum)\n-- [ ] `apply_event` writes to durable collections (nodes, scopes) and cursor in one RocksDB `WriteBatch`\n-- [ ] Journal + Reddit import paths use same apply\n-- [ ] Batched writes where possible (one flush per vote / per import batch)\n-- [ ] Idempotency tests: replay/catch-up never double-applies vote edge weights\n-- [ ] Tests: apply event → read back from durable\n-\n-### Phase 2 — Stop full tree at boot\n-\n-- [ ] Startup: open durable, catch up from projection cursor, then serve\n-- [ ] No `GlobalTree::new()` + full replay into RAM on normal boot\n-- [ ] `replay-index` command / flag: stream JSONL → durable (offline rebuild)\n-- [ ] Recovery tests: crash after JSONL append, crash after projection write, corrupt/missing cursor\n-- [ ] Integration tests use replay-index fixture or temp DB\n-\n-### Phase 3 — Scope load on read\n-\n-- [ ] `ScopeView { parent, children, group }` loaded from durable\n-- [ ] HTML / vote / pair / fetch handlers take `ScopeView` instead of `&GlobalTree`\n-- [ ] Remove or shrink `Arc<RwLock<GlobalTree>>`\n-\n-### Phase 4 — Scope cache\n-\n-- [ ] LRU with byte budget for hot scopes\n-- [ ] Invalidate on write to that scope\n-- [ ] Metrics: cache hit/miss, evictions, scope load time\n-\n-### Phase 5 — Durable API polish (optional)\n-\n-- [ ] Reified path/mutation API + `WriteBatch` integration in `durable`\n-- [ ] RocksDB tuning preset for 256MB Fly (`block_cache`, write buffers)\n-- [ ] Document `replay-index` in AGENTS.md\n-\n----\n-\n-## Non-goals (for now)\n-\n-- Distributed replication or multi-writer\n-- Replacing JSONL as canonical store\n-- SQL query layer over state\n-- Materialized rank scores on disk (unless RC latency forces it)\n-- Switching from RocksDB to sled without benchmarks\n-\n----\n-\n-## Open questions\n-\n-1. **One DB or two?** `{data_dir}/entity_db` today vs single `{data_dir}/store` — merge on Phase 1?\n-2. **Event identity** — add event sequence numbers now, or start with byte offset + checksum?\n-3. **Append durability mode** — pay fsync cost per mutation/batch, or document an acknowledged data-loss window?\n-4. **Scope key encoding** — string `ItemId` paths vs hashed; must match event `scope` field.\n-5. **Child scope wiring** — Reddit `apply_entity_under_parent` creates children without full path ensure; durable schema must preserve this.\n-6. **Clojure smoke tests** — still read `events.jsonl`; durable is internal. No change expected.\n-7. **Fly volume** — `/data` holds JSONL + RocksDB; monitor disk alongside RAM.\n-\n----\n-\n-## Summary\n-\n-**JSONL = write log. Durable = full reducer on disk + fat payloads. RAM = one rank-centrality graph (or a small LRU of scopes).**\n-\n-Option B from design discussions is not separate from “reducer in durable” — it **is** reducer in durable, with JSONL still owning writes and recovery. The incremental work is: projection on apply, drop full tree at boot, scope load on read, then cache with eviction.\n","role":"user"}],"model":"openai/gpt-chat-latest"}