Commit B is a precise, correct bugfix: it moves the zero-ratio guard before ensure_item/voted_pairs.insert so ghost items and phantom voted-pairs no longer pollute reducer state, and it updates the test to assert the corrected invariant. Commit A is a larger feature/refactor (faster scan, ingest replay CLI) that adds useful tooling but is more diffuse, higher risk of new bugs, and less critical than fixing genuine state-corruption in the core reducer.
constitution · epochs · watch · epoch 3
comparison
c_6209cd238b3f (tommy-mor) vs c_abd68b5e771a (tommy-mor)
download prompt · raw event · cmp_369514ecc721ab
council reasoning
B fixes a real reducer correctness bug: zero-ratio votes were rejected only after ensure_item and voted_pairs.insert, leaving ghost items and false voted pairs; moving the guard before side effects (with an updated test) protects ranking state integrity. A is solid tooling work—fast parse-only scan, richer parse_error surfaces, and compile --ingest—but it improves offline ergonomics rather than core simulation correctness, so it trails slightly on lasting project value.
Side B fixes a correctness bug by moving the zero-ratio early return before any side effects, preventing ghost items from being indexed and voted pairs from being recorded incorrectly. The accompanying test is updated to verify that no items, edges, or voted pairs are registered, whereas Side A is primarily a tooling enhancement (faster scan mode, richer parse errors, and new CLI functionality) rather than a core state-correctness fix.
sides
A — c_6209cd238b3f (tommy-mor)
message
[6e344666] Improve sorterc scan speed, errors, and ingest compile. Make scan a fast DSL parse pass with human-readable output, surface full parse_error details, and add compile --ingest for single-event replay from a log. Co-authored-by: Cursor <cursoragent@cursor.com>
diff preview
diff --git a/server/src/offline.rs b/server/src/offline.rs
index 54ad0ded096a305ef8454ab2cdd1c3af71b14f5d..2db6f67e22e00a24ce673d13095644dd9fa9342d 100644
--- a/server/src/offline.rs
+++ b/server/src/offline.rs
@@ -28,6 +28,10 @@ pub struct CompileResult {
pub threads: Vec<String>,
pub rankings: Vec<CheckScopeRanking>,
pub stats: CompileStats,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub ingest_id: Option<String>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub ingest_line: Option<usize>,
}
#[derive(Debug, Clone, Serialize)]
@@ -36,6 +40,8 @@ pub struct CompileError {
pub error: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub hint: Option<String>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub parse_error: Option<String>,
}
#[derive(Debug, Clone, Serialize)]
@@ -50,7 +56,7 @@ pub struct MalformedIngest {
pub id: String,
pub room_id: String,
pub thread_tag: String,
- pub reason: String,
+ pub parse_error: String,
}
#[derive(Debug, Clone, Serialize)]
@@ -59,9 +65,36 @@ pub struct ScanResult {
pub path: String,
pub total_lines: usize,
pub parsed_events: usize,
+ pub ingest_events: usize,
pub bad_json_lines: Vec<BadJsonLine>,
pub malformed_ingests: Vec<MalformedIngest>,
- pub skipped_ingests: usize,
+}
+
+#[derive(Debug)]
+pub enum CompileIngestError {
+ NotFound(String),
+ Io(std::io::Error),
+ Compile(CompileError),
+}
+
+impl CompileIngestError {
+ pub fn into_compile_error(self) -> CompileError {
+ match self {
+ Self::NotFound(id) => CompileError {
+ ok: false,
+ error: format!("ingest not found: {id}"),
+ hint: Some("pass the ingest event id from events.jsonl".into()),
+ parse_error: None,
+ },
+ Self::Io(e) => CompileError {
+ ok: false,
+ error: format!("io error: {e}"),
+ hint: None,
+ parse_error: None,
+ },
+ Self::Compile(e) => e,
+ }
+ }
}
fn document_stats(doc: &dsl::Document) -> CompileStats {
@@ -166,19 +199,22 @@ fn rankings_for_simulated(
.collect()
}
-/// Validate and simulate one `.sorter` document against optional base reducer state.
-pub fn compile_document(
+fn compile_document_inner(
base: &ReducerState,
room: &str,
text: &str,
+ ingest_id: Option<String>,
+ ingest_line: Option<usize>,
) -> Result<CompileResult, CompileError> {
let room_key = room.trim();
let scope = scope_from_room_wire(room_key);
let validated = validate_ingest_document(base, text, &scope).map_err(|(_, message, hint)| {
+ let parse_error = dsl::parse_full(text).err().map(|e| e.to_string());
CompileError {
ok: false,
error: message,
hint,
+ parse_error,
}
})?;
@@ -200,15 +236,27 @@ pub fn compile_document(
threads: threads_in_document(text),
rankings: rankings_for_simulated(&simulated, &scope, room_key, &validated.doc),
stats: document_stats(&validated.doc),
+ ingest_id,
+ ingest_line,
})
}
+/// Validate and simulate one `.sorter` document against optional base reducer state.
+pub fn compile_document(
+ base: &ReducerState,
+ room: &str,
+ text: &str,
+) -> Result<CompileResult, CompileError> {
+ compile_document_inner(base, room, text, None, None)
+}
+
fn ingest_parse_error(raw: &str) -> Option<String> {
dsl::parse_full(raw).err().map(|e| e.to_string())
}
-fn load_events_from_jsonl(path: &Path) -> Result<(Vec<(usize, Event)>, Vec<BadJsonLine>), std::io::Error> {
+fn load_events_from_jsonl(path: &Path) -> Result<(usize, Vec<(usize, Event)>, Vec<BadJsonLine>), std::io::Error> {
let text = std::fs::read_to_string(path)?;
+ let total_lines = text.lines().count();
let mut events = Vec::new();
let mut bad_json_lines = Vec::new();
for (idx, line) in text.lines().enumerate() {
@@ -225,61 +273,90 @@ fn load_events_from_jsonl(path: &Path) -> Result<(Vec<(usize, Event)>, Vec<BadJs
}),
}
}
- Ok((events, bad_json_lines))
+ Ok((total_lines, events, bad_json_lines))
}
-/// Replay a JSONL event log into reducer state (same rules as server boot).
-pub fn load_reducer_from_jsonl(path: &Path) -> Result<(ReducerState, Vec<BadJsonLine>), std::io::Error> {
- let (events, bad_json_lines) = load_events_from_jsonl(path)?;
+fn replay_events(events: &[(usize, Event)]) -> ReducerState {
let mut state = ReducerState::default();
for (_line_no, ev) in events {
- state.apply_event(ev);
+ state.apply_event(ev.clone());
}
- Ok((state, bad_json_lines))
+ state
}
-/// Scan an events.jsonl for corrupt JSON lines and ingests that fail DSL replay.
-pub fn scan_jsonl(path: &Path) -> Result<ScanResult, std::io::Error> {
- let text = std::fs::read_to_string(path)?;
- let total_lines = text.lines().count();
- let (events, bad_json_lines) = load_events_from_jsonl(path)?;
+/// Replay a JSONL event log into reducer state (same rules as server boot).
+pub fn load_reducer_from_jsonl(path: &Path) -> Result<(ReducerState, Vec<BadJsonLine>), std::io::Error> {
+ let (_total_lines, events, bad_json_lines) = load_events_from_jsonl(path)?;
+ Ok((replay_events(&events), bad_json_lines))
+}
- let mut malformed_ingests = Vec::new();
- let mut skipped_ingests = 0usize;
- let mut state = ReducerState::default();
- let parsed_events = events.len();
+/// Find one ingest in a log and compile it against all prior events as base state.
+pub fn compile_ingest_from_log(path: &Path, ingest_id: &str) -> Result<CompileResult, CompileIngestError> {
+ let (_total_lines, events, bad_json_lines) = load_events_from_jsonl(path).map_err(CompileIngestError::Io)?;
+ if !bad_json_lines.is_empty() {
+ return Err(CompileIngestError::Compile(CompileError {
+ ok: false,
+ error: format!("jsonl has {} corrupt line(s)", bad_json_lines.len()),
+ hint: Some("fix the log or use `sorterc scan`".into()),
+ parse_error: None,
+ }));
+ }
+
+ let needle = ingest_id.trim();
+ let mut found: Option<(usize, Ingest)> = None;
+ let mut prior: Vec<(usize, Event)> = Vec::new();
for (line_no, ev) in events {
if let Event::Ingest(ref ing) = ev {
- if let Some(reason) = ingest_parse_error(&ing.raw) {
+ if ing.id == needle {
+ found = Some((line_no, ing.clone()));
+ break;
+ }
+ }
+ prior.push((line_no, ev));
+ }
+
+ let (line_no, ing) = found.ok_or_else(|| CompileIngestError::NotFound(needle.to_string()))?;
+ let base = replay_events(&prior);
+ compile_document_inner(&base, &ing.room_id, &ing.raw, Some(ing.id.clone()), Some(line_no))
+ .map_err(CompileIngestError::Compile)
+}
+
+/// Scan an events.jsonl for corrupt JSON lines and ingests whose DSL fails to parse.
+///
+/// This does not replay the log (which would run rank centrality on every ingest and
+/// can take minutes on real logs). It matches what the server skips on boot: parse failure.
+pub fn scan_jsonl(path: &Path) -> Result<ScanResult, std::io::Error> {
+ let (total_lines, events, bad_json_lines) = load_events_from_jsonl(path)?;
+
+ let mut malformed_ingests = Vec::new();
+ let mut ingest_events = 0usize;
+
+ for (line_no, ev) in &events {
+ if let Event::Ingest(ing) = ev {
+ ingest_events += 1;
+ if let Some(parse_error) = ingest_parse_error(&ing.raw) {
malformed_ingests.push(MalformedIngest {
- line: line_no,
+ line: *line_no,
id: ing.id.clone(),
room_id: ing.room_id.clone(),
thread_tag: ing.thread_tag.clone(),
- reason,
+ parse_error,
});
}
- let before = state.ingests_by_id.len();
- state.apply_event(ev);
- if state.ingests_by_id.len() == before {
- skipped_ingests += 1;
- }
- } else {
- state.apply_event(ev);
}
}
- let ok = bad_json_lines.is_empty() && malformed_ingests.is_empty() && skipped_ingests == 0;
+ let ok = bad_json_lines.is_empty() && malformed_ingests.is_empty();
Ok(ScanResult {
ok,
path: path.display().to_string(),
total_lines,
- parsed_events,
+ parsed_events: events.len(),
+ ingest_events,
bad_json_lines,
malformed_ingests,
- skipped_ingests,
})
}
@@ -330,4 +407,25 @@ mod tests {
assert!(!report.ok);
assert_eq!(report.bad_json_lines.len(), 1);
}
+
+ #[test]
+ fn scan_reports_dsl_parse_error_detail() {
+ let dir = tempfile::tempdir().unwrap();
+ let path = dir.path().join("events.jsonl");
+ let ingest = serde_json::json!({
+ "type": "ingest",
+ "ts": 1,
+ "id": "bad-ingest-id",
+ "raw": "{ no closing brace\n~/a { body }\n~/a 1:0 ~/b",
+ "principal": "test",
+ "room_id": "public",
+ "thread_tag": "t",
+ });
+ std::fs::write(&path, format!("{ingest}\n")).unwrap();
+ let report = scan_jsonl(&path).unwrap();
+ assert!(!report.ok);
+ assert_eq!(report.malformed_ingests.len(), 1);
+ assert_eq!(report.malformed_ingests[0].id, "bad-ingest-id");
+ assert!(report.malformed_ingests[0].parse_error.contains("parse error"));
+ }
}
diff --git a/sorterc/readme.md b/sorterc/readme.md
index 1ebcc3fc935541ea9e47e0458ec67a750fe19fff..e388615c34b40914ca51fc7a79cb738eebc2909b 100644
--- a/sorterc/readme.md
+++ b/sorterc/readme.md
@@ -60,21 +60,33 @@ Rankings use the same structure as the server's dry-run check: parent scope, con
### `scan` — lint an `events.jsonl`
-Reads a JSONL event log and reports problems without starting a server.
+Fast single-pass check. Does **not** replay the log (replay runs rank centrality on every ingest and gets slow fast).
```bash
cargo run -p sorterc -- scan events.jsonl
-cargo run -p sorterc -- scan events.jsonl --pretty
+cargo run -p sorterc -- scan events.jsonl --json
+cargo run -p sorterc -- scan events.jsonl --json --pretty
```
+Default output is human-readable with a blank line between each problem. Use `--json` for machine output.
+
Reports:
- **bad JSON lines** — lines that are not valid JSON
-- **malformed ingests** — ingest events whose `raw` DSL fails to parse
-- **skipped ingests** — ingests dropped during replay (same behavior as server boot)
+- **malformed ingests** — ingest events whose `raw` DSL fails to parse, with full `parse_error` text
Exits 0 when clean, 1 when any issue is found.
+### `compile --ingest` — compile one event from a log
+
+Replay all events **before** the target ingest as base state, then compile that ingest's DSL:
+
+```bash
+cargo run -p sorterc -- compile --ingest cabd8adc-57ae-402d-a940-8e24339ac451 --from events.jsonl --pretty
+```
+
+Output includes `ingest_id`, `ingest_line`, and rankings for that post only. This may take a while for ingests late in a large log (full replay up to that point).
+
## Typical uses
- Iterate on `.sorter` files in an editor and pipe through `compile` to see rankings instantly
diff --git a/sorterc/src/main.rs b/sorterc/src/main.rs
index 71382c180085cb0ad71043c852f8db5d3a48a284..5687c850d16973d28749380b8dc89f3bcebbfc56 100644
--- a/sorterc/src/main.rs
+++ b/sorterc/src/main.rs
@@ -22,9 +22,15 @@ struct Cli {
enum Command {
/// Parse and simulate a .sorter document; emit ranking JSON to stdout.
Compile {
- /// `.sorter` file, or `-` f
… preview truncated; 6,030 characters omittedB — c_abd68b5e771a (tommy-mor)
message
[81de487b] Fix zero-ratio guard in reducer to drop before registering items or pair. Previously the early-return for zero-weight votes happened after ensure_item and voted_pairs.insert, leaving ghost items in the index and the pair incorrectly marked as voted. Move the check to before any side effects. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
diff preview
diff --git a/server/src/reducer.rs b/server/src/reducer.rs
index 6841d35cfc9de2389f340a22b8a45acb335e36c3..0e36979abe0f051493038ff7e652efc7f7a0ac80 100644
--- a/server/src/reducer.rs
+++ b/server/src/reducer.rs
@@ -112,6 +112,10 @@ impl GroupState {
if vote.ratio_right < 0 {
vote.ratio_right = 0;
}
+ if vote.ratio_left == 0 || vote.ratio_right == 0 {
+ // Zero on either side produces no valid edge; drop before registering items or pair.
+ return;
+ }
let a_idx = self.ensure_item(&vote.a);
let b_idx = self.ensure_item(&vote.b);
@@ -121,10 +125,6 @@ impl GroupState {
let w_a = vote.ratio_left as f64;
let w_b = vote.ratio_right as f64;
- if w_a == 0.0 || w_b == 0.0 {
- // Zero on either side produces no valid edge; drop the vote.
- return;
- }
self.add_edge_weight(b_idx, a_idx, w_a);
self.add_edge_weight(a_idx, b_idx, w_b);
diff --git a/server/tests/basic.rs b/server/tests/basic.rs
index cc8c1a0d139f3722ba6ecd13dd001c65be835b67..08159f4a7f0850fd165817a4a1af4f31ced2ad76 100644
--- a/server/tests/basic.rs
+++ b/server/tests/basic.rs
@@ -546,12 +546,10 @@ fn reducer_negative_ratio_clamped_to_zero() {
delegate: Some("00000000-0000-0000-0000-000000000000:test:local/test".to_string()),
thread_tag: "t".to_string(),
});
- // Items are registered, but the zero-clamped vote produces no edges.
- assert_eq!(group.idx_to_item.len(), 2);
- let a_idx = group.item_to_idx[&item_id("https://slug.social/~/t/a")];
- let b_idx = group.item_to_idx[&item_id("https://slug.social/~/t/b")];
- assert!(!group.edges.contains_key(&(a_idx, b_idx)));
- assert!(!group.edges.contains_key(&(b_idx, a_idx)));
+ // Nothing registered: zero-clamped vote is dropped before ensure_item.
+ assert!(group.idx_to_item.is_empty());
+ assert!(group.edges.is_empty());
+ assert!(group.voted_pairs.is_empty());
}
Hardlinks — judgments / attempts / prompt
judgments
attempts
Prompt text is loaded only by the download route.