Side A converts a synchronous fetch action into a proper SSE streaming flow with progress states, error handling, retry/backoff via a oneshot result channel, and improved diagnostic logging in the reddit worker, while also cleaning up module structure (moving fetch UI code into its own module) — a real behavioral improvement with matching test updates. Side B adds a standalone dev CLI (sorterc) that reuses existing server logic for offline compile/lint; it's useful tooling but is additive/peripheral, mostly wrapping existing validate/ranking code without changing core system behavior or fixing bugs.
constitution · epochs · watch · epoch 3
comparison
c_06fce70179bc (tommy-mor) vs c_0c466eea1ed4 (tommy-mor)
download prompt · raw event · cmp_bb52a4842b9ab5
council reasoning
A redesigns a core user path: Reddit entity fetch becomes an awaited SSE stream with oneshot job results, progressive UI morphs, and richer failure handling in the worker—lasting product architecture—versus B’s solid but peripheral offline compile/scan binary. A’s gain is tempered by extra logging and some removed unit tests, so the margin is meaningful but not huge.
Side A introduces a substantive architectural change: entity fetching is converted from a fire-and-forget JS response into an SSE-based workflow with progress/completion events, coordinated via oneshot notifications from the Reddit worker, plus the necessary client-side stream handling and refactoring of fetch-related HTML into its own module. Side B adds a useful developer-only offline tool (`sorterc`) for compiling `.sorter` files and linting `events.jsonl`, but it primarily extends tooling rather than changing the runtime behavior of the application, so its long-term project impact is somewhat narrower.
sides
A — c_06fce70179bc (tommy-mor)
message
[6d04afc2] refactor
diff preview
diff --git a/Cargo.lock b/Cargo.lock
index 2cea973082716e761ef6f5dd5886acc08ff9aac0..8c43fb75c472b102e6e1d3b837dce3355be898f2 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -17,6 +17,28 @@ version = "1.0.102"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c"
+[[package]]
+name = "async-stream"
+version = "0.3.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0b5a71a6f37880a80d1d7f19efd781e4b5de42c88f0722cc13bcb6cc2cfe8476"
+dependencies = [
+ "async-stream-impl",
+ "futures-core",
+ "pin-project-lite",
+]
+
+[[package]]
+name = "async-stream-impl"
+version = "0.3.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c7c24de15d275a1ecfd47a380fb4d5ec9bfe0933f309ed5e705b775596a3574d"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
[[package]]
name = "async-trait"
version = "0.1.89"
@@ -1242,9 +1264,11 @@ dependencies = [
name = "sorter2-server"
version = "0.0.1"
dependencies = [
+ "async-stream",
"axum",
"axum-extra",
"dotenvy",
+ "futures-util",
"maud",
"reqwest",
"serde",
diff --git a/server/Cargo.toml b/server/Cargo.toml
index bd600138b613bd0f546bdec217a5334cdcb20aa5..c940acb687fb141d21760a3d6656172013cf6f41 100644
--- a/server/Cargo.toml
+++ b/server/Cargo.toml
@@ -18,6 +18,8 @@ tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
reqwest = { version = "0.12", features = ["json"] }
dotenvy = "0.15"
+async-stream = "0.3"
+futures-util = { version = "0.3", default-features = false, features = ["std"] }
[dev-dependencies]
reqwest = { version = "0.12", features = ["json"] }
diff --git a/server/src/api/ui_html.rs b/server/src/api/ui_html.rs
index b33a84e8bb5e817b26592868d88090e6d664d950..7af6527d03c483f33f3469ce6766c01a554c5fe3 100644
--- a/server/src/api/ui_html.rs
+++ b/server/src/api/ui_html.rs
@@ -6,7 +6,8 @@ use axum::{
use std::collections::HashMap;
use crate::{
- html::{entity_section, input_panel, js_string_literal, ranking_panel, JsBuilder},
+ fetch,
+ html::{input_panel, js_string_literal, ranking_panel, JsBuilder},
parser::parse_reddit_url,
path_types::ItemId,
reddit::ensure_partial_tree,
@@ -89,18 +90,8 @@ pub async fn post_ui_html(
},
HtmlUiAction::FetchEntity { item } => {
let id = parse_item_param(&item);
- if id.is_root() {
- return ui_js_warn("nothing to fetch for the root").into_response();
- }
- state.queue_entity_fetch(id.clone());
- let tree = state.tree.read().await;
- let empty = crate::reducer::NodeState::default();
- let node = tree.get(&id).unwrap_or(&empty);
- let panel = entity_section(&id, node, true);
- JsBuilder::new()
- .morph_selector("#entity-section", panel)
- .into_response()
- },
+ fetch::fetch_entity_stream(state, id).into_response()
+ }
}
}
diff --git a/server/src/fetch/html.rs b/server/src/fetch/html.rs
new file mode 100644
index 0000000000000000000000000000000000000000..63634508496e224c38b9ec0308b7a6086462f925
--- /dev/null
+++ b/server/src/fetch/html.rs
@@ -0,0 +1,67 @@
+//! Markup for entity import / “Fetch from Reddit” (`POST /ui`, SSE response).
+
+use maud::{html, Markup};
+
+use crate::{
+ form_template::template_json_compact,
+ path_types::ItemId,
+ reddit::is_fetchable,
+ reducer::NodeState,
+ ui_action::UI_RPC_FIELD,
+};
+
+fn entity_panel(node: &NodeState) -> Markup {
+ html! {
+ @if let Some(data) = &node.data {
+ div id="entity-panel" class="entity-card" {
+ h2 { (data.title) }
+ @if let Some(author) = &data.author {
+ p class="muted small" { "by " (author) }
+ }
+ @if let Some(body) = &data.body_html {
+ div class="entity-body" { (maud::PreEscaped(body)) }
+ }
+ }
+ }
+ }
+}
+
+/// Reddit/API import — `POST /ui` with `fetch_entity` returns an SSE stream.
+pub fn fetch_entity_panel(item: &ItemId, has_data: bool, fetching: bool) -> Markup {
+ if !is_fetchable(item) {
+ return html! {};
+ }
+ let label = if fetching {
+ "Fetching…"
+ } else if has_data {
+ "Fetch more"
+ } else {
+ "Fetch from Reddit"
+ };
+ let rpc = template_json_compact(&serde_json::json!({
+ "action": "fetch_entity",
+ "item": item.as_str(),
+ }))
+ .expect("fetch_entity rpc template");
+ html! {
+ form method="post" action="/ui" id="fetch-entity-form" class="fetch-entity-form" {
+ input type="hidden" name=(UI_RPC_FIELD) value=(rpc);
+ @if fetching {
+ button type="submit" class="btn-secondary" disabled { (label) }
+ } @else {
+ button type="submit" class="btn-secondary" { (label) }
+ }
+ }
+ }
+}
+
+/// Entity card + fetch control (target `#entity-section` for Idiomorph / SSE).
+pub fn entity_section(item: &ItemId, node: &NodeState, fetching: bool) -> Markup {
+ let has_data = node.data.is_some();
+ html! {
+ section id="entity-section" class="demo-panel" {
+ (entity_panel(node))
+ (fetch_entity_panel(item, has_data, fetching))
+ }
+ }
+}
diff --git a/server/src/fetch/mod.rs b/server/src/fetch/mod.rs
new file mode 100644
index 0000000000000000000000000000000000000000..2290f9d3a0f1cbf1806c6339f82a4515c11cc3d3
--- /dev/null
+++ b/server/src/fetch/mod.rs
@@ -0,0 +1,115 @@
+//! Entity import over `POST /ui` as SSE (Reddit worker in [`crate::reddit`]).
+
+pub mod html;
+
+use std::convert::Infallible;
+use std::time::Duration;
+
+use async_stream::stream;
+use axum::response::sse::{Event, KeepAlive, Sse};
+use futures_util::Stream;
+use serde::Serialize;
+use tokio::sync::oneshot;
+
+use crate::{
+ path_types::ItemId,
+ reddit::FetchJobResult,
+ reducer::NodeState,
+ state::AppState,
+};
+
+pub fn now_ms() -> i64 {
+ let t = std::time::SystemTime::now()
+ .duration_since(std::time::UNIX_EPOCH)
+ .unwrap_or_default();
+ t.as_millis() as i64
+}
+
+#[derive(Serialize)]
+struct SseMorphPayload {
+ selector: &'static str,
+ html: String,
+}
+
+fn morph_complete_event(html: maud::Markup) -> Event {
+ let payload = SseMorphPayload {
+ selector: "#entity-section",
+ html: html.into_string(),
+ };
+ let data = serde_json::to_string(&payload).unwrap_or_else(|_| "{}".into());
+ Event::default().event("complete").data(data)
+}
+
+/// Stream `fetching` → `complete` / `error` for [`crate::ui_action::HtmlUiAction::FetchEntity`].
+pub fn fetch_entity_stream(
+ state: AppState,
+ id: ItemId,
+) -> Sse<impl Stream<Item = Result<Event, Infallible>>> {
+ tracing::debug!(item = %id, "fetch entity stream opened");
+
+ let stream = stream! {
+ if id.is_root() {
+ yield Ok(Event::default().event("error").data("{\"message\":\"nothing to fetch for the root\"}"));
+ return;
+ }
+
+ if !crate::reddit::is_fetchable(&id) {
+ tracing::debug!(item = %id, "fetch stream: not fetchable");
+ yield Ok(Event::default().event("error").data("{\"message\":\"this page cannot be fetched from Reddit\"}"));
+ return;
+ }
+
+ let fetching_html = {
+ let tree = state.tree.read().await;
+ let empty = NodeState::default();
+ let node = tree.get(&id).unwrap_or(&empty);
+ html::entity_section(&id, node, true).into_string()
+ };
+ let fetching_payload = serde_json::json!({
+ "selector": "#entity-section",
+ "html": fetching_html,
+ });
+ yield Ok(Event::default().event("fetching").data(fetching_payload.to_string()));
+
+ let (tx, rx) = oneshot::channel();
+ state.reddit.request_fetch(id.clone(), true, Some(tx));
+ tracing::debug!(item = %id, "fetch stream: queued reddit job");
+
+ let result = match rx.await {
+ Ok(r) => r,
+ Err(_) => {
+ tracing::warn!(item = %id, "fetch stream: worker dropped oneshot");
+ FetchJobResult::Failed("reddit worker stopped".into())
+ }
+ };
+
+ tracing::debug!(item = %id, ?result, "fetch stream: job finished");
+
+ match result {
+ FetchJobResult::Imported | FetchJobResult::NotFound => {
+ let tree = state.tree.read().await;
+ let empty = NodeState::default();
+ let node = tree.get(&id).unwrap_or(&empty);
+ yield Ok(morph_complete_event(html::entity_section(&id, node, false)));
+ }
+ FetchJobResult::SkippedCached | FetchJobResult::SkippedDuplicate => {
+ let tree = state.tree.read().await;
+ let empty = NodeState::default();
+ let node = tree.get(&id).unwrap_or(&empty);
+ yield Ok(morph_complete_event(html::entity_section(&id, node, false)));
+ }
+ FetchJobResult::RateLimited { reset_secs } => {
+ yield Ok(Event::default().event("error").data(
+ serde_json::json!({"message": format!("Reddit rate limit — retry in {reset_secs}s")}).to_string(),
+ ));
+ }
+ FetchJobResult::Failed(msg) => {
+ yield Ok(Event::default().event("error").data(
+ serde_json::json!({"message": msg}).to_string(),
+ ));
+ }
+ }
+ };
+
+ Sse::new(stream).keep_alive(KeepAlive::new().interval(Duration::from_secs(15)))
+}
diff --git a/server/src/html/mod.rs b/server/src/html/mod.rs
index db5b4c7f06b0be64603981166835cde268234f67..9314a7556306ddab969b896dbf4126b542a46722 100644
--- a/server/src/html/mod.rs
+++ b/server/src/html/mod.rs
@@ -7,10 +7,10 @@ use axum::{
use maud::{html, Markup, DOCTYPE};
use crate::{
+ fetch::html::entity_section,
form_template::template_json_compact,
path_types::ItemId,
ranking::{top_bottom, RankedItem},
- reddit::is_fetchable,
reducer::{GroupState, NodeState},
state::AppState,
ui_action::UI_RPC_FIELD,
@@ -149,62 +149,6 @@ pub fn breadcrumb_path(item: &ItemId) -> Markup {
}
}
-fn entity_panel(node: &NodeState) -> Markup {
- html! {
- @if let Some(data) = &node.data {
- div id="entity-panel" class="entity-card" {
- h2 { (data.title) }
- @if let Some(author) = &data.author {
- p class="muted small" { "by " (author) }
- }
- @if let Some(body) = &data.body_html {
- div class="entity-body" { (maud::PreEscaped(body)) }
- }
- }
- }
- }
-}
-
-/// Reddit/API import control — only shown on fetchable pages; never auto-fires.
-pub fn fetch_entity_panel(item: &ItemId, has_data: bool, fetching: bool) -> Markup {
- if !is_fetchable(item) {
- return html! {};
- }
- let label = if fetching {
- "Fetching…"
- } else if has_data {
- "Fetch more"
- } else {
- "Fetch from Reddit"
- };
- let rpc = template_json_compact(&serde_json::json!({
- "action": "fetch_entity",
- "item": item.as_str(),
- }))
- .expect("fetch_entity rpc template");
- html! {
- form method="post" action="/ui" id="fetch-entity-form" class="fetch-entity-form" {
- input type="hidden" name=(UI_RPC_FIELD) value=(rpc);
- @if fetching {
- button type="submit" class="btn-secondary" disabled { (label) }
- } @else {
- button type="submit" class="btn-secondary" { (label) }
- }
- }
- }
-}
-
-/// Entity
… preview truncated; 22,618 characters omittedB — c_0c466eea1ed4 (tommy-mor)
message
[9e20d06c] Add sorterc dev tool for offline DSL compile and JSONL lint. Introduce a workspace-only binary that validates .sorter files into ranking JSON and scans events.jsonl for corrupt or unreplayable ingests. Co-authored-by: Cursor <cursoragent@cursor.com>
diff preview
diff --git a/Cargo.lock b/Cargo.lock
index bf8153d9c723af97122c9ffdd4a7cfe82e853bb6..a07734f089b466440c3ae6fc1087ce85fc24ce62 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -1826,6 +1826,17 @@ dependencies = [
"windows-sys 0.60.2",
]
+[[package]]
+name = "sorterc"
+version = "0.0.1"
+dependencies = [
+ "anyhow",
+ "clap",
+ "serde",
+ "serde_json",
+ "slugsocial-server",
+]
+
[[package]]
name = "spin"
version = "0.9.8"
diff --git a/Cargo.toml b/Cargo.toml
index 149cbf07901eab57c593184ff8719a75d530f1da..25337acdd61e44b20f354c78fed4a88caf896280 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,5 +1,5 @@
[workspace]
-members = ["server", "cli"]
+members = ["server", "cli", "sorterc"]
resolver = "2"
diff --git a/agents.md b/agents.md
index d8b801e454fdf37e7ac6038b91a69f83b0746d59..ce646ed3cd7123be4732dccec4a6800467e651e7 100644
--- a/agents.md
+++ b/agents.md
@@ -93,6 +93,17 @@ SLUG_GOOGLE_CLIENT_SECRET=mock
After OAuth completes, the pending-session poll returns a `slug_…` bearer token for API calls.
+### Dev-only offline tooling
+
+**`sorterc`** — workspace binary, not published via npm. Compiles `.sorter` files and lints `events.jsonl` without a server:
+
+```
+cargo run -p sorterc -- compile path/to/doc.sorter [--base events.jsonl] [--room public] [--pretty]
+cargo run -p sorterc -- scan path/to/events.jsonl [--pretty]
+```
+
+`compile` validates DSL, simulates ingest against empty (or `--base`) reducer state, and prints JSON rankings. `scan` reports corrupt JSONL lines and ingests that fail DSL replay.
+
### Testing
- **Rust tests:** `cargo nextest run --workspace` (163 tests; requires `cargo-nextest`)
diff --git a/server/src/lib.rs b/server/src/lib.rs
index c1d477d21aea03aff00e6f0689b0b4379d0d68d2..ad8e31099c807fb5844acb16cd5086a2f19327a7 100644
--- a/server/src/lib.rs
+++ b/server/src/lib.rs
@@ -10,6 +10,7 @@ pub mod form_template;
pub mod html;
pub mod identity;
pub mod middleware;
+pub mod offline;
pub mod path_types;
pub mod ranking;
pub mod reducer;
diff --git a/server/src/offline.rs b/server/src/offline.rs
new file mode 100644
index 0000000000000000000000000000000000000000..54ad0ded096a305ef8454ab2cdd1c3af71b14f5d
--- /dev/null
+++ b/server/src/offline.rs
@@ -0,0 +1,333 @@
+//! Offline `.sorter` compilation and JSONL diagnostics (no network, no auth).
+
+use std::collections::HashSet;
+use std::path::Path;
+
+use serde::Serialize;
+use slug_types::{CheckScopeRanking, RankComponent, RankRow, paths::GardenItemUrl};
+
+use crate::{
+ api::{resolve_item, validate_ingest_document},
+ dsl,
+ events::{Event, Ingest},
+ path_types::ItemId,
+ reducer::{ReducerState, ScopeId, scope_from_room_wire},
+ scope_rank::build_children_rankings,
+};
+
+#[derive(Debug, Clone, Serialize)]
+pub struct CompileStats {
+ pub items: usize,
+ pub votes: usize,
+ pub prose_blocks: usize,
+}
+
+#[derive(Debug, Serialize)]
+pub struct CompileResult {
+ pub ok: bool,
+ pub threads: Vec<String>,
+ pub rankings: Vec<CheckScopeRanking>,
+ pub stats: CompileStats,
+}
+
+#[derive(Debug, Clone, Serialize)]
+pub struct CompileError {
+ pub ok: bool,
+ pub error: String,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub hint: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize)]
+pub struct BadJsonLine {
+ pub line: usize,
+ pub message: String,
+}
+
+#[derive(Debug, Clone, Serialize)]
+pub struct MalformedIngest {
+ pub line: usize,
+ pub id: String,
+ pub room_id: String,
+ pub thread_tag: String,
+ pub reason: String,
+}
+
+#[derive(Debug, Clone, Serialize)]
+pub struct ScanResult {
+ pub ok: bool,
+ pub path: String,
+ pub total_lines: usize,
+ pub parsed_events: usize,
+ pub bad_json_lines: Vec<BadJsonLine>,
+ pub malformed_ingests: Vec<MalformedIngest>,
+ pub skipped_ingests: usize,
+}
+
+fn document_stats(doc: &dsl::Document) -> CompileStats {
+ let mut items = 0usize;
+ let mut votes = 0usize;
+ let mut prose_blocks = 0usize;
+ for stmt in &doc.statements {
+ match stmt {
+ dsl::Stmt::Item { .. } => items += 1,
+ dsl::Stmt::Vote { .. } => votes += 1,
+ dsl::Stmt::Prose { .. } => prose_blocks += 1,
+ }
+ }
+ CompileStats {
+ items,
+ votes,
+ prose_blocks,
+ }
+}
+
+fn threads_in_document(text: &str) -> Vec<String> {
+ let mut out = HashSet::new();
+ for line in text.lines() {
+ let trimmed = line.trim();
+ if !trimmed.starts_with('#') {
+ continue;
+ }
+ let rest = trimmed.trim_start_matches('#').trim();
+ if rest.is_empty() {
+ continue;
+ }
+ let tag = rest.split_whitespace().next().unwrap_or(rest);
+ let tag = tag.split(':').next().unwrap_or(tag).trim();
+ if tag.is_empty() {
+ continue;
+ }
+ out.insert(format!("#{}", crate::canonical_path::canonicalize_tag(tag)));
+ }
+ let mut tags: Vec<String> = out.into_iter().collect();
+ tags.sort();
+ tags
+}
+
+fn voted_parent_scopes(doc: &dsl::Document) -> Vec<ItemId> {
+ let mut parents = HashSet::new();
+ for stmt in &doc.statements {
+ if let dsl::Stmt::Vote { item1, item2, .. } = stmt {
+ if let (Ok(a), Ok(b)) = (resolve_item(item1), resolve_item(item2)) {
+ if let Some(p) = a.parent() {
+ parents.insert(p);
+ }
+ if let Some(p) = b.parent() {
+ parents.insert(p);
+ }
+ }
+ }
+ }
+ let mut out: Vec<ItemId> = parents.into_iter().collect();
+ out.sort();
+ out
+}
+
+fn rankings_for_simulated(
+ simulated: &ReducerState,
+ scope: &ScopeId,
+ room_wire: &str,
+ doc: &dsl::Document,
+) -> Vec<CheckScopeRanking> {
+ voted_parent_scopes(doc)
+ .iter()
+ .map(|parent| {
+ let scoped_content = simulated
+ .content_for_scope(&scope)
+ .unwrap_or_else(|| simulated.public());
+ let scoped = build_children_rankings(scoped_content, parent);
+ let components: Vec<RankComponent> = scoped
+ .component_rankings
+ .into_iter()
+ .map(|comp| RankComponent {
+ pairs: comp.pairs,
+ ranking: comp
+ .ranked
+ .into_iter()
+ .map(|r| RankRow {
+ item: GardenItemUrl::from_stored(&r.item, room_wire),
+ score: r.score,
+ percent: None,
+ })
+ .collect(),
+ })
+ .collect();
+ CheckScopeRanking {
+ parent: GardenItemUrl::from_stored(parent, room_wire).into_inner(),
+ components,
+ unranked_items: scoped
+ .unranked_items
+ .into_iter()
+ .map(|it| GardenItemUrl::from_stored(&it, room_wire))
+ .collect(),
+ }
+ })
+ .collect()
+}
+
+/// Validate and simulate one `.sorter` document against optional base reducer state.
+pub fn compile_document(
+ base: &ReducerState,
+ room: &str,
+ text: &str,
+) -> Result<CompileResult, CompileError> {
+ let room_key = room.trim();
+ let scope = scope_from_room_wire(room_key);
+ let validated = validate_ingest_document(base, text, &scope).map_err(|(_, message, hint)| {
+ CompileError {
+ ok: false,
+ error: message,
+ hint,
+ }
+ })?;
+
+ let event = Event::Ingest(Ingest {
+ ts: validated.ts,
+ id: uuid::Uuid::new_v4().to_string(),
+ raw: validated.raw_text.clone(),
+ principal: "offline".to_string(),
+ delegate: None,
+ room_id: room_key.to_string(),
+ thread_tag: "offline".to_string(),
+ });
+
+ let mut simulated = base.clone();
+ simulated.apply_event(event);
+
+ Ok(CompileResult {
+ ok: true,
+ threads: threads_in_document(text),
+ rankings: rankings_for_simulated(&simulated, &scope, room_key, &validated.doc),
+ stats: document_stats(&validated.doc),
+ })
+}
+
+fn ingest_parse_error(raw: &str) -> Option<String> {
+ dsl::parse_full(raw).err().map(|e| e.to_string())
+}
+
+fn load_events_from_jsonl(path: &Path) -> Result<(Vec<(usize, Event)>, Vec<BadJsonLine>), std::io::Error> {
+ let text = std::fs::read_to_string(path)?;
+ let mut events = Vec::new();
+ let mut bad_json_lines = Vec::new();
+ for (idx, line) in text.lines().enumerate() {
+ let line_no = idx + 1;
+ let trimmed = line.trim();
+ if trimmed.is_empty() {
+ continue;
+ }
+ match serde_json::from_str::<Event>(trimmed) {
+ Ok(ev) => events.push((line_no, ev)),
+ Err(e) => bad_json_lines.push(BadJsonLine {
+ line: line_no,
+ message: e.to_string(),
+ }),
+ }
+ }
+ Ok((events, bad_json_lines))
+}
+
+/// Replay a JSONL event log into reducer state (same rules as server boot).
+pub fn load_reducer_from_jsonl(path: &Path) -> Result<(ReducerState, Vec<BadJsonLine>), std::io::Error> {
+ let (events, bad_json_lines) = load_events_from_jsonl(path)?;
+ let mut state = ReducerState::default();
+ for (_line_no, ev) in events {
+ state.apply_event(ev);
+ }
+ Ok((state, bad_json_lines))
+}
+
+/// Scan an events.jsonl for corrupt JSON lines and ingests that fail DSL replay.
+pub fn scan_jsonl(path: &Path) -> Result<ScanResult, std::io::Error> {
+ let text = std::fs::read_to_string(path)?;
+ let total_lines = text.lines().count();
+ let (events, bad_json_lines) = load_events_from_jsonl(path)?;
+
+ let mut malformed_ingests = Vec::new();
+ let mut skipped_ingests = 0usize;
+ let mut state = ReducerState::default();
+ let parsed_events = events.len();
+
+ for (line_no, ev) in events {
+ if let Event::Ingest(ref ing) = ev {
+ if let Some(reason) = ingest_parse_error(&ing.raw) {
+ malformed_ingests.push(MalformedIngest {
+ line: line_no,
+ id: ing.id.clone(),
+ room_id: ing.room_id.clone(),
+ thread_tag: ing.thread_tag.clone(),
+ reason,
+ });
+ }
+ let before = state.ingests_by_id.len();
+ state.apply_event(ev);
+ if state.ingests_by_id.len() == before {
+ skipped_ingests += 1;
+ }
+ } else {
+ state.apply_event(ev);
+ }
+ }
+
+ let ok = bad_json_lines.is_empty() && malformed_ingests.is_empty() && skipped_ingests == 0;
+
+ Ok(ScanResult {
+ ok,
+ path: path.display().to_string(),
+ total_lines,
+ parsed_events,
+ bad_json_lines,
+ malformed_ingests,
+ skipped_ingests,
+ })
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ const TUTORIAL: &str = include_str!("../tests/fixtures/tutorial.sorter");
+
+ #[test]
+ fn compile_tutorial_fixture_emits_rankings() {
+ let result = compile_document(&ReducerState::default(), "public", TUTORIAL).unwrap();
+ assert!(result.ok);
+ assert!(!result.threads.is_empty());
+ assert!(result.stats.items >= 6);
+ assert!(result.stats.votes >= 6);
+ assert!(!result.rankings.is_empty());
+ }
+
+ #[test]
+ fn compile_rejects_vote_on_missing_item() {
+ let err = compile_document(
+ &ReducerState::default(),
+ "public",
+ "{ reason }\n~/missing/a 2:1 ~/missing/b",
+ )
+ .unwrap_err();
+ assert!(!err.ok);
+ assert!(err.error.contains("undefined"));
+ }
+
+ #[test]
+ fn scan
… preview truncated; 7,950 characters omittedHardlinks — judgments / attempts / prompt
judgments
attempts
Prompt text is loaded only by the download route.