{"messages":[{"content":"You are a constitutional council ranking individual git commits for ownership allocation.\n\nCompare these two commits. Decide which contributed more lasting value to the project.\n\nJudge substance, not spectacle:\n- Prefer correct, lasting design and real bugfixes over churn, formatting, renames, or generated noise.\n- Prefer clarity and necessity over sheer line count. A small precise change can beat a large diffuse one.\n- Do not favor a side merely because its patch is longer or noisier.\n- Weight what the change does for the project, not the contributor's name.\n\nReturn ONLY a JSON object: {\"winner\": \"A\" or \"B\", \"ratio\": \"N:M\", \"explanation\": \"...\"}\nThe explanation must cite concrete differences in the patches (1-3 sentences).\n\nSide A — contributor: tommy-mor\nSide A — commit message:\n[6d04afc2] refactor\n\nSide A — unified diff (full patch):\ndiff --git a/Cargo.lock b/Cargo.lock\nindex 2cea973082716e761ef6f5dd5886acc08ff9aac0..8c43fb75c472b102e6e1d3b837dce3355be898f2 100644\n--- a/Cargo.lock\n+++ b/Cargo.lock\n@@ -17,6 +17,28 @@ version = \"1.0.102\"\n source = \"registry+https://github.com/rust-lang/crates.io-index\"\n checksum = \"7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c\"\n \n+[[package]]\n+name = \"async-stream\"\n+version = \"0.3.6\"\n+source = \"registry+https://github.com/rust-lang/crates.io-index\"\n+checksum = \"0b5a71a6f37880a80d1d7f19efd781e4b5de42c88f0722cc13bcb6cc2cfe8476\"\n+dependencies = [\n+ \"async-stream-impl\",\n+ \"futures-core\",\n+ \"pin-project-lite\",\n+]\n+\n+[[package]]\n+name = \"async-stream-impl\"\n+version = \"0.3.6\"\n+source = \"registry+https://github.com/rust-lang/crates.io-index\"\n+checksum = \"c7c24de15d275a1ecfd47a380fb4d5ec9bfe0933f309ed5e705b775596a3574d\"\n+dependencies = [\n+ \"proc-macro2\",\n+ \"quote\",\n+ \"syn\",\n+]\n+\n [[package]]\n name = \"async-trait\"\n version = \"0.1.89\"\n@@ -1242,9 +1264,11 @@ dependencies = [\n name = \"sorter2-server\"\n version = \"0.0.1\"\n dependencies = [\n+ \"async-stream\",\n  \"axum\",\n  \"axum-extra\",\n  \"dotenvy\",\n+ \"futures-util\",\n  \"maud\",\n  \"reqwest\",\n  \"serde\",\ndiff --git a/server/Cargo.toml b/server/Cargo.toml\nindex bd600138b613bd0f546bdec217a5334cdcb20aa5..c940acb687fb141d21760a3d6656172013cf6f41 100644\n--- a/server/Cargo.toml\n+++ b/server/Cargo.toml\n@@ -18,6 +18,8 @@ tracing = \"0.1\"\n tracing-subscriber = { version = \"0.3\", features = [\"env-filter\"] }\n reqwest = { version = \"0.12\", features = [\"json\"] }\n dotenvy = \"0.15\"\n+async-stream = \"0.3\"\n+futures-util = { version = \"0.3\", default-features = false, features = [\"std\"] }\n \n [dev-dependencies]\n reqwest = { version = \"0.12\", features = [\"json\"] }\ndiff --git a/server/src/api/ui_html.rs b/server/src/api/ui_html.rs\nindex b33a84e8bb5e817b26592868d88090e6d664d950..7af6527d03c483f33f3469ce6766c01a554c5fe3 100644\n--- a/server/src/api/ui_html.rs\n+++ b/server/src/api/ui_html.rs\n@@ -6,7 +6,8 @@ use axum::{\n use std::collections::HashMap;\n \n use crate::{\n-    html::{entity_section, input_panel, js_string_literal, ranking_panel, JsBuilder},\n+    fetch,\n+    html::{input_panel, js_string_literal, ranking_panel, JsBuilder},\n     parser::parse_reddit_url,\n     path_types::ItemId,\n     reddit::ensure_partial_tree,\n@@ -89,18 +90,8 @@ pub async fn post_ui_html(\n         },\n         HtmlUiAction::FetchEntity { item } => {\n             let id = parse_item_param(&item);\n-            if id.is_root() {\n-                return ui_js_warn(\"nothing to fetch for the root\").into_response();\n-            }\n-            state.queue_entity_fetch(id.clone());\n-            let tree = state.tree.read().await;\n-            let empty = crate::reducer::NodeState::default();\n-            let node = tree.get(&id).unwrap_or(&empty);\n-            let panel = entity_section(&id, node, true);\n-            JsBuilder::new()\n-                .morph_selector(\"#entity-section\", panel)\n-                .into_response()\n-        },\n+            fetch::fetch_entity_stream(state, id).into_response()\n+        }\n     }\n }\n \ndiff --git a/server/src/fetch/html.rs b/server/src/fetch/html.rs\nnew file mode 100644\nindex 0000000000000000000000000000000000000000..63634508496e224c38b9ec0308b7a6086462f925\n--- /dev/null\n+++ b/server/src/fetch/html.rs\n@@ -0,0 +1,67 @@\n+//! Markup for entity import / “Fetch from Reddit” (`POST /ui`, SSE response).\n+\n+use maud::{html, Markup};\n+\n+use crate::{\n+    form_template::template_json_compact,\n+    path_types::ItemId,\n+    reddit::is_fetchable,\n+    reducer::NodeState,\n+    ui_action::UI_RPC_FIELD,\n+};\n+\n+fn entity_panel(node: &NodeState) -> Markup {\n+    html! {\n+        @if let Some(data) = &node.data {\n+            div id=\"entity-panel\" class=\"entity-card\" {\n+                h2 { (data.title) }\n+                @if let Some(author) = &data.author {\n+                    p class=\"muted small\" { \"by \" (author) }\n+                }\n+                @if let Some(body) = &data.body_html {\n+                    div class=\"entity-body\" { (maud::PreEscaped(body)) }\n+                }\n+            }\n+        }\n+    }\n+}\n+\n+/// Reddit/API import — `POST /ui` with `fetch_entity` returns an SSE stream.\n+pub fn fetch_entity_panel(item: &ItemId, has_data: bool, fetching: bool) -> Markup {\n+    if !is_fetchable(item) {\n+        return html! {};\n+    }\n+    let label = if fetching {\n+        \"Fetching…\"\n+    } else if has_data {\n+        \"Fetch more\"\n+    } else {\n+        \"Fetch from Reddit\"\n+    };\n+    let rpc = template_json_compact(&serde_json::json!({\n+        \"action\": \"fetch_entity\",\n+        \"item\": item.as_str(),\n+    }))\n+    .expect(\"fetch_entity rpc template\");\n+    html! {\n+        form method=\"post\" action=\"/ui\" id=\"fetch-entity-form\" class=\"fetch-entity-form\" {\n+            input type=\"hidden\" name=(UI_RPC_FIELD) value=(rpc);\n+            @if fetching {\n+                button type=\"submit\" class=\"btn-secondary\" disabled { (label) }\n+            } @else {\n+                button type=\"submit\" class=\"btn-secondary\" { (label) }\n+            }\n+        }\n+    }\n+}\n+\n+/// Entity card + fetch control (target `#entity-section` for Idiomorph / SSE).\n+pub fn entity_section(item: &ItemId, node: &NodeState, fetching: bool) -> Markup {\n+    let has_data = node.data.is_some();\n+    html! {\n+        section id=\"entity-section\" class=\"demo-panel\" {\n+            (entity_panel(node))\n+            (fetch_entity_panel(item, has_data, fetching))\n+        }\n+    }\n+}\ndiff --git a/server/src/fetch/mod.rs b/server/src/fetch/mod.rs\nnew file mode 100644\nindex 0000000000000000000000000000000000000000..2290f9d3a0f1cbf1806c6339f82a4515c11cc3d3\n--- /dev/null\n+++ b/server/src/fetch/mod.rs\n@@ -0,0 +1,115 @@\n+//! Entity import over `POST /ui` as SSE (Reddit worker in [`crate::reddit`]).\n+\n+pub mod html;\n+\n+use std::convert::Infallible;\n+use std::time::Duration;\n+\n+use async_stream::stream;\n+use axum::response::sse::{Event, KeepAlive, Sse};\n+use futures_util::Stream;\n+use serde::Serialize;\n+use tokio::sync::oneshot;\n+\n+use crate::{\n+    path_types::ItemId,\n+    reddit::FetchJobResult,\n+    reducer::NodeState,\n+    state::AppState,\n+};\n+\n+pub fn now_ms() -> i64 {\n+    let t = std::time::SystemTime::now()\n+        .duration_since(std::time::UNIX_EPOCH)\n+        .unwrap_or_default();\n+    t.as_millis() as i64\n+}\n+\n+#[derive(Serialize)]\n+struct SseMorphPayload {\n+    selector: &'static str,\n+    html: String,\n+}\n+\n+fn morph_complete_event(html: maud::Markup) -> Event {\n+    let payload = SseMorphPayload {\n+        selector: \"#entity-section\",\n+        html: html.into_string(),\n+    };\n+    let data = serde_json::to_string(&payload).unwrap_or_else(|_| \"{}\".into());\n+    Event::default().event(\"complete\").data(data)\n+}\n+\n+/// Stream `fetching` → `complete` / `error` for [`crate::ui_action::HtmlUiAction::FetchEntity`].\n+pub fn fetch_entity_stream(\n+    state: AppState,\n+    id: ItemId,\n+) -> Sse<impl Stream<Item = Result<Event, Infallible>>> {\n+    tracing::debug!(item = %id, \"fetch entity stream opened\");\n+\n+    let stream = stream! {\n+        if id.is_root() {\n+            yield Ok(Event::default().event(\"error\").data(\"{\\\"message\\\":\\\"nothing to fetch for the root\\\"}\"));\n+            return;\n+        }\n+\n+        if !crate::reddit::is_fetchable(&id) {\n+            tracing::debug!(item = %id, \"fetch stream: not fetchable\");\n+            yield Ok(Event::default().event(\"error\").data(\"{\\\"message\\\":\\\"this page cannot be fetched from Reddit\\\"}\"));\n+            return;\n+        }\n+\n+        let fetching_html = {\n+            let tree = state.tree.read().await;\n+            let empty = NodeState::default();\n+            let node = tree.get(&id).unwrap_or(&empty);\n+            html::entity_section(&id, node, true).into_string()\n+        };\n+        let fetching_payload = serde_json::json!({\n+            \"selector\": \"#entity-section\",\n+            \"html\": fetching_html,\n+        });\n+        yield Ok(Event::default().event(\"fetching\").data(fetching_payload.to_string()));\n+\n+        let (tx, rx) = oneshot::channel();\n+        state.reddit.request_fetch(id.clone(), true, Some(tx));\n+        tracing::debug!(item = %id, \"fetch stream: queued reddit job\");\n+\n+        let result = match rx.await {\n+            Ok(r) => r,\n+            Err(_) => {\n+                tracing::warn!(item = %id, \"fetch stream: worker dropped oneshot\");\n+                FetchJobResult::Failed(\"reddit worker stopped\".into())\n+            }\n+        };\n+\n+        tracing::debug!(item = %id, ?result, \"fetch stream: job finished\");\n+\n+        match result {\n+            FetchJobResult::Imported | FetchJobResult::NotFound => {\n+                let tree = state.tree.read().await;\n+                let empty = NodeState::default();\n+                let node = tree.get(&id).unwrap_or(&empty);\n+                yield Ok(morph_complete_event(html::entity_section(&id, node, false)));\n+            }\n+            FetchJobResult::SkippedCached | FetchJobResult::SkippedDuplicate => {\n+                let tree = state.tree.read().await;\n+                let empty = NodeState::default();\n+                let node = tree.get(&id).unwrap_or(&empty);\n+                yield Ok(morph_complete_event(html::entity_section(&id, node, false)));\n+            }\n+            FetchJobResult::RateLimited { reset_secs } => {\n+                yield Ok(Event::default().event(\"error\").data(\n+                    serde_json::json!({\"message\": format!(\"Reddit rate limit — retry in {reset_secs}s\")}).to_string(),\n+                ));\n+            }\n+            FetchJobResult::Failed(msg) => {\n+                yield Ok(Event::default().event(\"error\").data(\n+                    serde_json::json!({\"message\": msg}).to_string(),\n+                ));\n+            }\n+        }\n+    };\n+\n+    Sse::new(stream).keep_alive(KeepAlive::new().interval(Duration::from_secs(15)))\n+}\ndiff --git a/server/src/html/mod.rs b/server/src/html/mod.rs\nindex db5b4c7f06b0be64603981166835cde268234f67..9314a7556306ddab969b896dbf4126b542a46722 100644\n--- a/server/src/html/mod.rs\n+++ b/server/src/html/mod.rs\n@@ -7,10 +7,10 @@ use axum::{\n use maud::{html, Markup, DOCTYPE};\n \n use crate::{\n+    fetch::html::entity_section,\n     form_template::template_json_compact,\n     path_types::ItemId,\n     ranking::{top_bottom, RankedItem},\n-    reddit::is_fetchable,\n     reducer::{GroupState, NodeState},\n     state::AppState,\n     ui_action::UI_RPC_FIELD,\n@@ -149,62 +149,6 @@ pub fn breadcrumb_path(item: &ItemId) -> Markup {\n     }\n }\n \n-fn entity_panel(node: &NodeState) -> Markup {\n-    html! {\n-        @if let Some(data) = &node.data {\n-            div id=\"entity-panel\" class=\"entity-card\" {\n-                h2 { (data.title) }\n-                @if let Some(author) = &data.author {\n-                    p class=\"muted small\" { \"by \" (author) }\n-                }\n-                @if let Some(body) = &data.body_html {\n-                    div class=\"entity-body\" { (maud::PreEscaped(body)) }\n-                }\n-            }\n-        }\n-    }\n-}\n-\n-/// Reddit/API import control — only shown on fetchable pages; never auto-fires.\n-pub fn fetch_entity_panel(item: &ItemId, has_data: bool, fetching: bool) -> Markup {\n-    if !is_fetchable(item) {\n-        return html! {};\n-    }\n-    let label = if fetching {\n-        \"Fetching…\"\n-    } else if has_data {\n-        \"Fetch more\"\n-    } else {\n-        \"Fetch from Reddit\"\n-    };\n-    let rpc = template_json_compact(&serde_json::json!({\n-        \"action\": \"fetch_entity\",\n-        \"item\": item.as_str(),\n-    }))\n-    .expect(\"fetch_entity rpc template\");\n-    html! {\n-        form method=\"post\" action=\"/ui\" id=\"fetch-entity-form\" class=\"fetch-entity-form\" {\n-            input type=\"hidden\" name=(UI_RPC_FIELD) value=(rpc);\n-            @if fetching {\n-                button type=\"submit\" class=\"btn-secondary\" disabled { (label) }\n-            } @else {\n-                button type=\"submit\" class=\"btn-secondary\" { (label) }\n-            }\n-        }\n-    }\n-}\n-\n-/// Entity card + explicit fetch control (morphed as `#entity-section`).\n-pub fn entity_section(item: &ItemId, node: &NodeState, fetching: bool) -> Markup {\n-    let has_data = node.data.is_some();\n-    html! {\n-        section id=\"entity-section\" class=\"demo-panel\" {\n-            (entity_panel(node))\n-            (fetch_entity_panel(item, has_data, fetching))\n-        }\n-    }\n-}\n-\n fn rank_list(label: &str, items: &[RankedItem], start_rank: usize) -> Markup {\n     html! {\n         @if !items.is_empty() {\ndiff --git a/server/src/lib.rs b/server/src/lib.rs\nindex 79b173f391a96ae5d0d96fd656e1e8d2dd070d09..0677363e0ed21d244bfa065f7ec728549ddd9e50 100644\n--- a/server/src/lib.rs\n+++ b/server/src/lib.rs\n@@ -1,6 +1,7 @@\n pub mod api;\n pub mod event_log;\n pub mod events;\n+pub mod fetch;\n pub mod form_template;\n pub mod html;\n pub mod parser;\ndiff --git a/server/src/reddit.rs b/server/src/reddit.rs\nindex ff0f01e57b18af878eb5be3efc47204a7673589d..0e6ce32720951a58456852b67fb76a85dd9f4aad 100644\n--- a/server/src/reddit.rs\n+++ b/server/src/reddit.rs\n@@ -7,12 +7,12 @@ use std::time::{Duration, Instant};\n use reqwest::{header, Client, StatusCode};\n use serde::Deserialize;\n use serde_json::Value;\n-use tokio::sync::{mpsc, RwLock};\n+use tokio::sync::{mpsc, oneshot, RwLock};\n \n use crate::{\n     event_log::EventLog,\n     events::Event,\n-    html::now_ms,\n+    fetch::now_ms,\n     path_types::ItemId,\n     reducer::GlobalTree,\n };\n@@ -22,10 +22,21 @@ pub fn ensure_partial_tree(tree: &mut GlobalTree, id: &ItemId) {\n     tree.ensure_path(id);\n }\n \n+#[derive(Debug, Clone, PartialEq, Eq)]\n+pub enum FetchJobResult {\n+    Imported,\n+    NotFound,\n+    SkippedDuplicate,\n+    SkippedCached,\n+    RateLimited { reset_secs: u64 },\n+    Failed(String),\n+}\n+\n pub struct RedditCommand {\n     pub id: ItemId,\n     /// User-initiated fetch bypasses the in-memory \"recently fetched\" cache.\n     pub force: bool,\n+    pub done: Option<oneshot::Sender<FetchJobResult>>,\n }\n \n #[derive(Clone)]\n@@ -72,14 +83,29 @@ impl RedditBroker {\n             .build()\n             .expect(\"reqwest client\");\n \n+        tracing::debug!(\n+            api_base = %config.api_base,\n+            oauth_base = %config.oauth_base,\n+            oauth = config.creds.is_some(),\n+            \"reddit worker started\"\n+        );\n+\n         tokio::spawn(reddit_worker(rx, tree, event_log, client, config));\n \n         Self { tx }\n     }\n \n     /// Queue a fetch; drops when the channel is full (backpressure).\n-    pub fn request_fetch(&self, id: ItemId, force: bool) {\n-        let _ = self.tx.try_send(RedditCommand { id, force });\n+    pub fn request_fetch(\n+        &self,\n+        id: ItemId,\n+        force: bool,\n+        done: Option<oneshot::Sender<FetchJobResult>>,\n+    ) {\n+        match self.tx.try_send(RedditCommand { id: id.clone(), force, done }) {\n+            Ok(()) => tracing::debug!(item = %id, force, \"reddit fetch queued\"),\n+            Err(_) => tracing::warn!(item = %id, \"reddit fetch queue full, dropped\"),\n+        }\n     }\n }\n \n@@ -95,8 +121,6 @@ impl RedditApiConfig {\n }\n \n impl RedditCredentials {\n-    /// Reddit's OAuth docs call these \"client id\" and \"client secret\"; the app\n-    /// registration UI often labels them \"app id\" / \"app secret\" — same values.\n     fn from_env() -> Option<Self> {\n         let client_id = std::env::var(\"REDDIT_CLIENT_ID\")\n             .or_else(|_| std::env::var(\"REDDIT_APP_ID\"))\n@@ -128,12 +152,10 @@ pub fn default_user_agent() -> String {\n     })\n }\n \n-/// True when this node can be loaded from the Reddit JSON API.\n pub fn is_fetchable(id: &ItemId) -> bool {\n     !map_item_to_reddit_api(id, \"https://example.com\").is_empty()\n }\n \n-/// Derive UI-facing fields from a stored payload (Reddit-specific when under reddit.com).\n pub fn entity_view_from_payload(id: &ItemId, payload: &Value) -> Option<crate::reducer::EntityData> {\n     if id.as_str().starts_with(\"reddit.com\") {\n         return parse_reddit_view(id, payload);\n@@ -141,12 +163,17 @@ pub fn entity_view_from_payload(id: &ItemId, payload: &Value) -> Option<crate::r\n     None\n }\n \n-/// Apply a full API payload to the in-memory tree (view derived for known domains).\n pub fn apply_entity_import(tree: &mut GlobalTree, id: &ItemId, payload: Value) {\n     let view = entity_view_from_payload(id, &payload);\n     tree.apply_entity_raw(id, payload, view);\n }\n \n+fn notify(done: Option<oneshot::Sender<FetchJobResult>>, result: FetchJobResult) {\n+    if let Some(tx) = done {\n+        let _ = tx.send(result);\n+    }\n+}\n+\n async fn reddit_worker(\n     mut rx: mpsc::Receiver<RedditCommand>,\n     tree: Arc<RwLock<GlobalTree>>,\n@@ -168,15 +195,25 @@ async fn reddit_worker(\n         recently_fetched.retain(|_, t| now.duration_since(*t) < cache_ttl);\n \n         if in_flight.contains(&cmd.id) {\n+            tracing::debug!(item = %cmd.id, \"reddit fetch skipped: already in flight\");\n+            notify(cmd.done, FetchJobResult::SkippedDuplicate);\n             continue;\n         }\n         if !cmd.force && recently_fetched.contains_key(&cmd.id) {\n+            tracing::debug!(item = %cmd.id, \"reddit fetch skipped: recently fetched cache\");\n+            notify(cmd.done, FetchJobResult::SkippedCached);\n             continue;\n         }\n \n         in_flight.insert(cmd.id.clone());\n         let fetch_id = cmd.id.clone();\n+        let done = cmd.done;\n \n+        tracing::debug!(\n+            item = %fetch_id,\n+            delay_ms = current_delay.as_millis(),\n+            \"reddit fetch starting after delay\"\n+        );\n         tokio::time::sleep(current_delay).await;\n \n         if let Some(c) = &creds {\n@@ -184,8 +221,18 @@ async fn reddit_worker(\n         }\n \n         let token = oauth.as_ref().map(|t| t.access_token.as_str());\n+        if token.is_some() {\n+            tracing::debug!(item = %fetch_id, \"reddit fetch using OAuth bearer\");\n+        }\n+\n+        let fetch_base = if token.is_some() {\n+            &oauth_base\n+        } else {\n+            &api_base\n+        };\n+        let outcome = do_fetch(&client, fetch_base, &fetch_id, token).await;\n \n-        match do_fetch(&client, &api_base, &fetch_id, token).await {\n+        match outcome {\n             Ok(FetchOutcome::Payload(payload)) => {\n                 let ts = now_ms();\n                 let event = Event::EntityImported {\n@@ -193,30 +240,49 @@ async fn reddit_worker(\n                     ts,\n                     payload: payload.clone(),\n                 };\n-                if let Err(e) = event_log.append(&event).await {\n-                    tracing::warn!(\"event log append failed for {}: {}\", fetch_id, e);\n-                } else {\n-                    apply_entity_import(&mut *tree.write().await, &fetch_id, payload);\n-                    recently_fetched.insert(fetch_id.clone(), Instant::now());\n-                    current_delay = Duration::from_millis(600);\n+                tracing::debug!(\n+                    item = %fetch_id,\n+                    ts,\n+                    payload_keys = ?payload.as_object().map(|o| o.len()),\n+                    \"reddit fetch got JSON payload, appending event\"\n+                );\n+                match event_log.append(&event).await {\n+                    Err(e) => {\n+                        tracing::warn!(\n+                            item = %fetch_id,\n+                            err = %e,\n+                            \"reddit event log append failed\"\n+                        );\n+                        notify(done, FetchJobResult::Failed(e.to_string()));\n+                    }\n+                    Ok(()) => {\n+                        apply_entity_import(&mut *tree.write().await, &fetch_id, payload);\n+                        recently_fetched.insert(fetch_id.clone(), Instant::now());\n+                        current_delay = Duration::from_millis(600);\n+                        tracing::info!(item = %fetch_id, \"reddit entity imported\");\n+                        notify(done, FetchJobResult::Imported);\n+                    }\n                 }\n             }\n             Ok(FetchOutcome::NotFound) => {\n+                tracing::debug!(item = %fetch_id, \"reddit fetch: not found (no event written)\");\n                 recently_fetched.insert(fetch_id.clone(), Instant::now());\n+                notify(done, FetchJobResult::NotFound);\n             }\n             Ok(FetchOutcome::RateLimited { reset_secs }) => {\n-                let wait = Duration::from_secs(reset_secs.max(1));\n                 tracing::warn!(\n-                    \"Reddit rate limit for {}; sleeping {}s\",\n-                    fetch_id,\n-                    wait.as_secs()\n+                    item = %fetch_id,\n+                    reset_secs,\n+                    \"reddit rate limited\"\n                 );\n-                tokio::time::sleep(wait).await;\n+                tokio::time::sleep(Duration::from_secs(reset_secs.max(1))).await;\n                 current_delay = (current_delay * 2).min(Duration::from_secs(60));\n+                notify(done, FetchJobResult::RateLimited { reset_secs });\n             }\n             Err(e) => {\n-                tracing::warn!(\"Reddit fetch failed for {}: {}\", fetch_id, e);\n+                tracing::warn!(item = %fetch_id, err = %e, \"reddit fetch failed\");\n                 current_delay = (current_delay * 2).min(Duration::from_secs(60));\n+                notify(done, FetchJobResult::Failed(e));\n             }\n         }\n \n@@ -238,6 +304,7 @@ async fn ensure_oauth_token(\n ) -> Option<OAuthToken> {\n     if let Some(t) = existing {\n         if Instant::now() < t.expires_at - Duration::from_secs(60) {\n+            tracing::debug!(\"reddit OAuth token still valid\");\n             return Some(t);\n         }\n     }\n@@ -246,6 +313,7 @@ async fn ensure_oauth_token(\n         \"{}/api/v1/access_token\",\n         oauth_base.trim_end_matches('/')\n     );\n+    tracing::debug!(%url, \"reddit OAuth token request\");\n \n     let resp = client\n         .post(&url)\n@@ -257,13 +325,13 @@ async fn ensure_oauth_token(\n     let resp = match resp {\n         Ok(r) => r,\n         Err(e) => {\n-            tracing::warn!(\"Reddit OAuth token request failed: {e}\");\n+            tracing::warn!(\"reddit OAuth token request failed: {e}\");\n             return None;\n         }\n     };\n \n     if !resp.status().is_success() {\n-        tracing::warn!(\"Reddit OAuth token HTTP {}\", resp.status());\n+        tracing::warn!(\"reddit OAuth token HTTP {}\", resp.status());\n         return None;\n     }\n \n@@ -276,11 +344,12 @@ async fn ensure_oauth_token(\n     let body: TokenResponse = match resp.json().await {\n         Ok(b) => b,\n         Err(e) => {\n-            tracing::warn!(\"Reddit OAuth token parse failed: {e}\");\n+            tracing::warn!(\"reddit OAuth token parse failed: {e}\");\n             return None;\n         }\n     };\n \n+    tracing::debug!(expires_in = body.expires_in, \"reddit OAuth token acquired\");\n     Some(OAuthToken {\n         access_token: body.access_token,\n         expires_at: Instant::now() + Duration::from_secs(body.expires_in),\n@@ -295,35 +364,72 @@ async fn do_fetch(\n ) -> Result<FetchOutcome, String> {\n     let url = map_item_to_reddit_api(id, api_base);\n     if url.is_empty() {\n+        tracing::debug!(item = %id, \"reddit do_fetch: no API URL for item\");\n         return Ok(FetchOutcome::NotFound);\n     }\n \n+    tracing::debug!(item = %id, %url, bearer = bearer.is_some(), \"reddit HTTP GET\");\n+\n     let mut req = client.get(&url);\n     if let Some(token) = bearer {\n         req = req.bearer_auth(token);\n     }\n \n-    let resp = req.send().await.map_err(|e| e.to_string())?;\n+    let resp = req.send().await.map_err(|e| {\n+        tracing::debug!(item = %id, %url, err = %e, \"reddit HTTP transport error\");\n+        e.to_string()\n+    })?;\n+\n+    let status = resp.status();\n+    tracing::debug!(\n+        item = %id,\n+        %url,\n+        %status,\n+        remaining = ?rate_limit_remaining(&resp),\n+        reset = ?rate_limit_reset_secs(&resp),\n+        \"reddit HTTP response\"\n+    );\n \n-    if resp.status() == StatusCode::TOO_MANY_REQUESTS {\n+    if status == StatusCode::TOO_MANY_REQUESTS {\n         let reset = rate_limit_reset_secs(&resp);\n         return Ok(FetchOutcome::RateLimited { reset_secs: reset });\n     }\n \n-    if resp.status() == StatusCode::SERVICE_UNAVAILABLE {\n-        return Err(\"Reddit unavailable (503)\".to_string());\n+    if status == StatusCode::SERVICE_UNAVAILABLE {\n+        return Err(\"Reddit unavailable (503)\".into());\n     }\n \n-    if !resp.status().is_success() {\n+    if !status.is_success() {\n+        let body = resp.text().await.unwrap_or_default();\n+        tracing::debug!(\n+            item = %id,\n+            %status,\n+            body_len = body.len(),\n+            body_prefix = %body.chars().take(240).collect::<String>(),\n+            \"reddit non-success body\"\n+        );\n         return Ok(FetchOutcome::NotFound);\n     }\n \n     if rate_limit_remaining(&resp) == Some(0) {\n         let reset = rate_limit_reset_secs(&resp);\n+        tracing::debug!(item = %id, reset_secs = reset, \"reddit headers: rate limit exhausted\");\n         return Ok(FetchOutcome::RateLimited { reset_secs: reset });\n     }\n \n-    let payload: Value = resp.json().await.map_err(|e| e.to_string())?;\n+    let text = resp.text().await.map_err(|e| e.to_string())?;\n+    tracing::debug!(item = %id, bytes = text.len(), \"reddit response body received\");\n+\n+    let payload: Value = serde_json::from_str(&text).map_err(|e| {\n+        tracing::debug!(\n+            item = %id,\n+            err = %e,\n+            body_prefix = %text.chars().take(240).collect::<String>(),\n+            \"reddit JSON parse failed\"\n+        );\n+        format!(\"invalid JSON: {e}\")\n+    })?;\n+\n     Ok(FetchOutcome::Payload(payload))\n }\n \n@@ -344,7 +450,6 @@ fn rate_limit_reset_secs(resp: &reqwest::Response) -> u64 {\n         .unwrap_or(5)\n }\n \n-/// Map canonical item id to a Reddit JSON API URL under `api_base`.\n pub fn map_item_to_reddit_api(id: &ItemId, api_base: &str) -> String {\n     let path = id.as_str();\n     if !path.starts_with(\"reddit.com/\") && path != \"reddit.com\" {\n@@ -445,26 +550,6 @@ mod tests {\n             map_item_to_reddit_api(&id, \"https://www.reddit.com\"),\n             \"https://www.reddit.com/r/rust/about.json?raw_json=1\"\n         );\n-        assert_eq!(\n-            map_item_to_reddit_api(&id, \"http://127.0.0.1:9999\"),\n-            \"http://127.0.0.1:9999/r/rust/about.json?raw_json=1\"\n-        );\n-    }\n-\n-    #[test]\n-    fn map_post_url() {\n-        let id = ItemId::parse(\"reddit.com/r/amitheasshole/comments/1trnvdl\").unwrap();\n-        assert_eq!(\n-            map_item_to_reddit_api(&id, \"https://www.reddit.com\"),\n-            \"https://www.reddit.com/r/amitheasshole/comments/1trnvdl.json?raw_json=1\"\n-        );\n-    }\n-\n-    #[test]\n-    fn is_fetchable_reddit_sub() {\n-        let id = ItemId::parse(\"reddit.com/r/rust\").unwrap();\n-        assert!(is_fetchable(&id));\n-        assert!(!is_fetchable(&ItemId::opaque(\"example.com/x\")));\n     }\n \n     #[test]\n@@ -477,19 +562,5 @@ mod tests {\n         )\n         .unwrap();\n         assert_eq!(entity.title, \"The Rust Programming Language\");\n-        assert!(entity.body_html.as_ref().is_some_and(|b| b.contains(\"Rust\")));\n-    }\n-\n-    #[test]\n-    fn parse_post_fixture() {\n-        let json = r#\"[{\"kind\":\"Listing\",\"data\":{\"children\":[{\"kind\":\"t3\",\"data\":{\"title\":\"AITA\",\"author\":\"op\",\"selftext_html\":\"&lt;p&gt;hi&lt;/p&gt;\",\"thumbnail\":\"https://b.thumbs.redditmedia.com/x.jpg\"}}]}}]\"#;\n-        let v: Value = serde_json::from_str(json).unwrap();\n-        let entity = entity_view_from_payload(\n-            &ItemId::parse(\"reddit.com/r/x/comments/abc\").unwrap(),\n-            &v,\n-        )\n-        .unwrap();\n-        assert_eq!(entity.title, \"AITA\");\n-        assert_eq!(entity.author.as_deref(), Some(\"op\"));\n     }\n }\ndiff --git a/server/src/state.rs b/server/src/state.rs\nindex d71d1079a8f486cdab15795384aef0b81b32544d..e7ff9f663e45b5e168d6a3869948e3bd890966d4 100644\n--- a/server/src/state.rs\n+++ b/server/src/state.rs\n@@ -143,9 +143,9 @@ impl AppState {\n         Ok(())\n     }\n \n-    /// User-initiated Reddit/API import (via \"Fetch more\" — never on paste or navigate).\n-    pub fn queue_entity_fetch(&self, id: ItemId) {\n-        self.reddit.request_fetch(id, true);\n+    /// User-initiated Reddit/API import (SSE / fetch module only).\n+    pub fn queue_entity_fetch(&self, id: ItemId, done: Option<tokio::sync::oneshot::Sender<crate::reddit::FetchJobResult>>) {\n+        self.reddit.request_fetch(id, true, done);\n     }\n \n     pub async fn record_vote(\ndiff --git a/server/src/ui_action.rs b/server/src/ui_action.rs\nindex 3d6a49a2a3fb950752827efe1f5a049308f09baf..ef9ac873fc0442e0b960f144309c8e91124d7884 100644\n--- a/server/src/ui_action.rs\n+++ b/server/src/ui_action.rs\n@@ -26,7 +26,7 @@ pub enum HtmlUiAction {\n     ParseQuery {\n         query: String,\n     },\n-    /// Fetch upstream entity data for the current page (explicit user action only).\n+    /// Import entity data; `POST /ui` responds with `text/event-stream` (not JS).\n     FetchEntity {\n         item: String,\n     },\ndiff --git a/server/static/sorter_ui.js b/server/static/sorter_ui.js\nindex d9b016f197547f37ca4b2bcdd7ee6b673d0fb3f0..fe7bfc657c0fccae5c596d005b0e807abefec677 100644\n--- a/server/static/sorter_ui.js\n+++ b/server/static/sorter_ui.js\n@@ -1,5 +1,5 @@\n /**\n- * sorter2 web UI: fetch/eval for POST /ui. No product logic here.\n+ * sorter2 web UI: POST /ui returns JS (morph) or SSE (entity fetch).\n  */\n (function () {\n   function evalJs(js) {\n@@ -8,15 +8,98 @@\n     }\n   }\n \n+  function morphSelector(selector, html) {\n+    var el = document.querySelector(selector);\n+    if (el && typeof Idiomorph !== 'undefined') {\n+      Idiomorph.morph(el, html);\n+    }\n+  }\n+\n+  function handleSseEvent(eventType, data, form) {\n+    if (eventType === 'fetching' || eventType === 'complete') {\n+      try {\n+        var msg = JSON.parse(data);\n+        morphSelector(msg.selector || '#entity-section', msg.html);\n+      } catch (err) {\n+        console.warn('fetch morph parse', err);\n+      }\n+    }\n+    if (eventType === 'complete' || eventType === 'error') {\n+      var btn = form && form.querySelector('button[type=\"submit\"]');\n+      if (btn) btn.disabled = false;\n+    }\n+    if (eventType === 'error') {\n+      try {\n+        var err = JSON.parse(data);\n+        console.warn('fetch error:', err.message || data);\n+      } catch (_e) {\n+        console.warn('fetch error:', data);\n+      }\n+    }\n+  }\n+\n+  function consumeSseStream(response, form) {\n+    var reader = response.body.getReader();\n+    var decoder = new TextDecoder();\n+    var buffer = '';\n+    var eventType = '';\n+    var dataLines = [];\n+\n+    function dispatch() {\n+      if (!eventType && dataLines.length === 0) return;\n+      handleSseEvent(eventType || 'message', dataLines.join('\\n'), form);\n+      eventType = '';\n+      dataLines = [];\n+    }\n+\n+    function pump() {\n+      return reader.read().then(function (chunk) {\n+        if (chunk.done) {\n+          dispatch();\n+          return;\n+        }\n+        buffer += decoder.decode(chunk.value, { stream: true });\n+        var parts = buffer.split('\\n');\n+        buffer = parts.pop() || '';\n+        for (var i = 0; i < parts.length; i++) {\n+          var line = parts[i].replace(/\\r$/, '');\n+          if (line === '') {\n+            dispatch();\n+          } else if (line.indexOf('event:') === 0) {\n+            eventType = line.slice(6).trim();\n+          } else if (line.indexOf('data:') === 0) {\n+            dataLines.push(line.slice(5).trim());\n+          }\n+        }\n+        return pump();\n+      });\n+    }\n+\n+    return pump();\n+  }\n+\n   function postUiForm(form) {\n+    var btn = form.querySelector('button[type=\"submit\"]');\n+    if (form.id === 'fetch-entity-form' && btn) {\n+      btn.disabled = true;\n+    }\n     return fetch(form.action, {\n       method: 'POST',\n       body: new URLSearchParams(new FormData(form)),\n       headers: { 'Content-Type': 'application/x-www-form-urlencoded' },\n       credentials: 'same-origin',\n     }).then(function (resp) {\n-      return resp.text();\n-    }).then(evalJs);\n+      var ct = resp.headers.get('content-type') || '';\n+      if (ct.indexOf('text/event-stream') !== -1) {\n+        return consumeSseStream(resp, form);\n+      }\n+      return resp.text().then(evalJs);\n+    }).catch(function (err) {\n+      if (form.id === 'fetch-entity-form' && btn) {\n+        btn.disabled = false;\n+      }\n+      console.warn('POST /ui failed', err);\n+    });\n   }\n \n   function initSorterUi() {\ndiff --git a/test/reddit_import.clj b/test/reddit_import.clj\nindex 54edaedeef08718c8f184d6aa468d8e6415068c7..17f2ee39e1498b0b6bb6a9d8a7e35608b0bd1b32 100644\n--- a/test/reddit_import.clj\n+++ b/test/reddit_import.clj\n@@ -44,10 +44,12 @@\n             (do (Thread/sleep 200) (recur))\n             false))))))\n \n-(defn- curl-post-ui [base rpc-json]\n+(defn- curl-fetch-ui-sse [base item]\n   (process/shell {:out :string :err :string}\n-                 \"curl\" \"-sf\" \"-X\" \"POST\" (str base \"/ui\")\n-                 \"--data-urlencode\" (str \"__rpc__=\" rpc-json)))\n+                 \"curl\" \"-sfN\" \"--max-time\" \"20\"\n+                 \"-X\" \"POST\" (str base \"/ui\")\n+                 \"--data-urlencode\"\n+                 (str \"__rpc__={\\\"action\\\":\\\"fetch_entity\\\",\\\"item\\\":\\\"\" item \"\\\"}\")))\n \n (defn- wait-event-log [path ms]\n   (let [deadline (+ (System/currentTimeMillis) ms)]\n@@ -98,11 +100,12 @@\n                                               \"curl\" \"-sf\" browse-url))]\n               (is (str/includes? before \"Fetch from Reddit\"))\n               (is (not (str/includes? before \"The Rust Programming Language\")))\n-              (let [rpc \"{\\\"action\\\":\\\"fetch_entity\\\",\\\"item\\\":\\\"reddit.com/r/rust\\\"}\"\n-                    post (curl-post-ui app-base rpc)\n-                    log-path (str data-dir \"/events.jsonl\")]\n-                (is (zero? (:exit post)) \"fetch_entity POST succeeds\")\n-                (is (wait-event-log log-path 10000) \"event log written\")\n+              (let [log-path (str data-dir \"/events.jsonl\")\n+                    sse (curl-fetch-ui-sse app-base \"reddit.com/r/rust\")]\n+                (is (zero? (:exit sse)) \"POST /ui fetch_entity SSE succeeds\")\n+                (is (str/includes? (:out sse) \"event: complete\"))\n+                (is (str/includes? (:out sse) \"The Rust Programming Language\"))\n+                (is (wait-event-log log-path 2000) \"event log written\")\n                 (let [after (:out (process/shell {:out :string :err :string}\n                                                  \"curl\" \"-sf\" browse-url))\n                       log (slurp (io/file log-path))]\n\n\nSide B — contributor: tommy-mor\nSide B — commit message:\n[6da4a15e] note\n\nSide B — unified diff (full patch):\ndiff --git a/notes.tdsl b/notes.tdsl\nindex a05487df6c445d9b333421e1d7781f2587f01298..067951bfdaa10b9c450d23bd1095a68f769f1e81 100644\n--- a/notes.tdsl\n+++ b/notes.tdsl\n@@ -28,3 +28,5 @@ need consistent + in corener of section header that morphs/expands in add form.\n my json-encoded plan is not being used for the forms. i want that same plan for the morphable/expandable + signs that produce forms. form target encoded... there was a plan.md with this. plan2.md...\n \n new public thread form needs fewer words\n+\n+new thread errors don't propogate properly. no #errors div, it used to be more precise? maybe it should be more precise?\n","role":"user"}],"model":"openai/gpt-chat-latest"}