B delivers a real, tested improvement: it fixes a slow full-replay scan (avoiding minutes-long DSL simulation), surfaces detailed parse errors, adds a working `compile --ingest` feature with new tests, and updates docs to match. A merely adds scaffolding (Dockerfile, fly.toml, deps.edn) and two loose top-level .rs files (event_log.rs, views.rs) with no visible module wiring or tests, making its lasting value unclear and possibly dead code.
constitution · epochs · watch · epoch 3
comparison
c_16438843de8f (tommy-mor) vs c_6209cd238b3f (tommy-mor)
download prompt · raw event · cmp_51cbe661384b60
council reasoning
B lands a lasting design fix: scan becomes a fast DSL-only pass (no per-ingest centrality replay), surfaces full parse_error text, adds compile --ingest with prior-state replay, plus CLI/docs/tests. A mostly seeds ops/config (Dockerfile, fly.toml, deps.edn) and two small new modules (event_log, views) without the same targeted product improvement.
Side B makes the offline tooling substantially more useful and efficient by changing `scan_jsonl` to a fast parse-only pass, surfacing detailed DSL `parse_error` information, adding `compile --ingest` to replay state up to a single event, improving CLI behavior, and adding a regression test. Side A adds important infrastructure (Dockerfile/Fly config) plus new `EventLog` and `ViewStore` modules, but much of it is deployment support or isolated new utilities, whereas Side B directly improves core developer workflows and diagnostics with targeted functional changes.
sides
A — c_16438843de8f (tommy-mor)
message
[4cd0d15d] more seed
diff preview
diff --git a/Dockerfile b/Dockerfile
new file mode 100644
index 0000000000000000000000000000000000000000..9cb07c60cb0da063f747cfbf1b3b876ecb8ba03e
--- /dev/null
+++ b/Dockerfile
@@ -0,0 +1,34 @@
+# time 0.3.47+ requires Rust 1.88 (edition 2024)
+FROM rust:1.88-slim as builder
+
+WORKDIR /build
+
+RUN apt-get update && \
+ apt-get install -y pkg-config libssl-dev && \
+ rm -rf /var/lib/apt/lists/*
+
+# Copy source and build. (Keep it simple to avoid remote build cache oddities.)
+COPY . .
+RUN cargo build --release --package slugsocial-server
+
+FROM debian:bookworm-slim
+
+RUN apt-get update && \
+ apt-get install -y ca-certificates && \
+ rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
+COPY --from=builder /build/target/release/slugsocial-server /app/slugsocial-server
+
+# Create data directory for persistent volume
+RUN mkdir -p /data
+
+ENV SLUG_DATA_DIR=/data
+ENV SLUG_EVENT_LOG=/data/events.jsonl
+ENV PORT=8080
+
+EXPOSE 8080
+
+CMD ["/app/slugsocial-server"]
+
diff --git a/deps.edn b/deps.edn
new file mode 100644
index 0000000000000000000000000000000000000000..0bf892d44f491cb2313e01ae8a942c3097c52948
--- /dev/null
+++ b/deps.edn
@@ -0,0 +1,10 @@
+{:paths ["." "test"]
+ :deps {cheshire/cheshire {:mvn/version "5.13.0"}
+ http-kit/http-kit {:mvn/version "2.8.0"}
+ babashka/fs {:mvn/version "0.5.32"}
+ babashka/process {:mvn/version "0.6.25"}
+ com.blockether/spel {:mvn/version "0.7.11"}}
+ :aliases
+ {:kaocha {:extra-deps {lambdaisland/kaocha {:mvn/version "1.91.1392"}
+ lambdaisland/kaocha-junit-xml {:mvn/version "1.17.101"}}
+ :main-opts ["-m" "kaocha.runner"]}}}
diff --git a/event_log.rs b/event_log.rs
new file mode 100644
index 0000000000000000000000000000000000000000..eaae0d495e43a45d6590603892265a62cc92906e
--- /dev/null
+++ b/event_log.rs
@@ -0,0 +1,83 @@
+use std::path::{Path, PathBuf};
+
+use tokio::{
+ fs::{self, OpenOptions},
+ io::{AsyncBufReadExt, AsyncWriteExt, BufReader},
+};
+
+use crate::events::Event;
+
+#[derive(Debug, thiserror::Error)]
+pub enum EventLogError {
+ #[error("io error: {0}")]
+ Io(#[from] std::io::Error),
+ #[error("json error: {0}")]
+ Json(#[from] serde_json::Error),
+}
+
+#[derive(Debug, Clone)]
+pub struct EventLog {
+ path: PathBuf,
+}
+
+impl EventLog {
+ pub fn new(path: impl Into<PathBuf>) -> Self {
+ Self { path: path.into() }
+ }
+
+ pub fn path(&self) -> &Path {
+ &self.path
+ }
+
+ pub async fn ensure_parent_dir(&self) -> Result<(), EventLogError> {
+ if let Some(parent) = self.path.parent() {
+ fs::create_dir_all(parent).await?;
+ }
+ Ok(())
+ }
+
+ pub async fn append(&self, event: &Event) -> Result<(), EventLogError> {
+ self.ensure_parent_dir().await?;
+ let mut f: tokio::fs::File = OpenOptions::new()
+ .create(true)
+ .append(true)
+ .open(&self.path)
+ .await?;
+
+ let mut line = serde_json::to_string(event)?;
+ line.push('\n');
+ f.write_all(line.as_bytes()).await?;
+ f.flush().await?;
+ Ok(())
+ }
+
+ /// Load events from JSONL. Corrupt lines are skipped and returned as `(line_no, line)`.
+ pub async fn load_all(&self) -> Result<(Vec<Event>, Vec<(usize, String)>), EventLogError> {
+ if !fs::try_exists(&self.path).await? {
+ return Ok((vec![], vec![]));
+ }
+
+ let f = fs::File::open(&self.path).await?;
+ let mut reader = BufReader::new(f).lines();
+
+ let mut events = Vec::new();
+ let mut bad_lines = Vec::new();
+
+ let mut line_no: usize = 0;
+ while let Some(line) = reader.next_line().await? {
+ line_no += 1;
+ let trimmed = line.trim();
+ if trimmed.is_empty() {
+ continue;
+ }
+ match serde_json::from_str::<Event>(trimmed) {
+ Ok(ev) => events.push(ev),
+ Err(_) => bad_lines.push((line_no, line)),
+ }
+ }
+
+ Ok((events, bad_lines))
+ }
+}
+
+
diff --git a/fly.toml b/fly.toml
new file mode 100644
index 0000000000000000000000000000000000000000..bbb9345e527452db1d87a549213645c195eae5fc
--- /dev/null
+++ b/fly.toml
@@ -0,0 +1,42 @@
+app = "slugsocial"
+primary_region = "iad"
+
+[build]
+ dockerfile = "Dockerfile"
+
+[env]
+ SLUG_DATA_DIR = "/data"
+ SLUG_EVENT_LOG = "/data/events.jsonl"
+ PORT = "8080"
+
+[[services]]
+ internal_port = 8080
+ protocol = "tcp"
+
+ [[services.ports]]
+ port = 80
+ handlers = ["http"]
+ force_https = true
+
+ [[services.ports]]
+ port = 443
+ handlers = ["tls", "http"]
+
+ [services.concurrency]
+ type = "connections"
+ hard_limit = 1000
+ soft_limit = 500
+
+ [[services.http_checks]]
+ interval = "10s"
+ timeout = "2s"
+ grace_period = "5s"
+ method = "GET"
+ path = "/healthz"
+ protocol = "http"
+ tls_skip_verify = false
+
+[[mounts]]
+ source = "slugsocial_data"
+ destination = "/data"
+
diff --git a/views.rs b/views.rs
new file mode 100644
index 0000000000000000000000000000000000000000..d4f0ffc49475f014698b4da0de6f476884430813
--- /dev/null
+++ b/views.rs
@@ -0,0 +1,63 @@
+use std::{
+ collections::HashMap,
+ sync::{Arc, Mutex},
+};
+use tokio::sync::mpsc;
+
+type CountMap = Arc<Mutex<HashMap<String, u64>>>;
+
+#[derive(Clone)]
+pub struct ViewStore {
+ counts: CountMap,
+ flush_tx: mpsc::Sender<()>,
+}
+
+impl ViewStore {
+ pub fn new(json_path: &str) -> Self {
+ // Load existing counts from disk on startup (best-effort)
+ let initial: HashMap<String, u64> = std::fs::read_to_string(json_path)
+ .ok()
+ .and_then(|s| serde_json::from_str(&s).ok())
+ .unwrap_or_default();
+
+ let counts: CountMap = Arc::new(Mutex::new(initial));
+ let (flush_tx, mut flush_rx) = mpsc::channel::<()>(64);
+ let path = json_path.to_string();
+
+ let counts_for_writer = counts.clone();
+ tokio::spawn(async move {
+ while flush_rx.recv().await.is_some() {
+ while flush_rx.try_recv().is_ok() {}
+
+ let snapshot: HashMap<String, u64> = {
+ counts_for_writer.lock().unwrap().clone()
+ };
+
+ let path = path.clone();
+ let _ = tokio::task::spawn_blocking(move || {
+ if let Ok(json) = serde_json::to_string(&snapshot) {
+ let tmp = format!("{path}.tmp");
+ if std::fs::write(&tmp, &json).is_ok() {
+ let _ = std::fs::rename(&tmp, &path);
+ }
+ }
+ })
+ .await;
+ }
+ });
+
+ Self { counts, flush_tx }
+ }
+
+ pub fn increment(&self, path: String) {
+ {
+ let mut map = self.counts.lock().unwrap();
+ *map.entry(path).or_insert(0) += 1;
+ }
+ let _ = self.flush_tx.try_send(());
+ }
+
+ pub fn get_views(&self, path: &str) -> u64 {
+ self.counts.lock().unwrap().get(path).copied().unwrap_or(0)
+ }
+}
B — c_6209cd238b3f (tommy-mor)
message
[6e344666] Improve sorterc scan speed, errors, and ingest compile. Make scan a fast DSL parse pass with human-readable output, surface full parse_error details, and add compile --ingest for single-event replay from a log. Co-authored-by: Cursor <cursoragent@cursor.com>
diff preview
diff --git a/server/src/offline.rs b/server/src/offline.rs
index 54ad0ded096a305ef8454ab2cdd1c3af71b14f5d..2db6f67e22e00a24ce673d13095644dd9fa9342d 100644
--- a/server/src/offline.rs
+++ b/server/src/offline.rs
@@ -28,6 +28,10 @@ pub struct CompileResult {
pub threads: Vec<String>,
pub rankings: Vec<CheckScopeRanking>,
pub stats: CompileStats,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub ingest_id: Option<String>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub ingest_line: Option<usize>,
}
#[derive(Debug, Clone, Serialize)]
@@ -36,6 +40,8 @@ pub struct CompileError {
pub error: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub hint: Option<String>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub parse_error: Option<String>,
}
#[derive(Debug, Clone, Serialize)]
@@ -50,7 +56,7 @@ pub struct MalformedIngest {
pub id: String,
pub room_id: String,
pub thread_tag: String,
- pub reason: String,
+ pub parse_error: String,
}
#[derive(Debug, Clone, Serialize)]
@@ -59,9 +65,36 @@ pub struct ScanResult {
pub path: String,
pub total_lines: usize,
pub parsed_events: usize,
+ pub ingest_events: usize,
pub bad_json_lines: Vec<BadJsonLine>,
pub malformed_ingests: Vec<MalformedIngest>,
- pub skipped_ingests: usize,
+}
+
+#[derive(Debug)]
+pub enum CompileIngestError {
+ NotFound(String),
+ Io(std::io::Error),
+ Compile(CompileError),
+}
+
+impl CompileIngestError {
+ pub fn into_compile_error(self) -> CompileError {
+ match self {
+ Self::NotFound(id) => CompileError {
+ ok: false,
+ error: format!("ingest not found: {id}"),
+ hint: Some("pass the ingest event id from events.jsonl".into()),
+ parse_error: None,
+ },
+ Self::Io(e) => CompileError {
+ ok: false,
+ error: format!("io error: {e}"),
+ hint: None,
+ parse_error: None,
+ },
+ Self::Compile(e) => e,
+ }
+ }
}
fn document_stats(doc: &dsl::Document) -> CompileStats {
@@ -166,19 +199,22 @@ fn rankings_for_simulated(
.collect()
}
-/// Validate and simulate one `.sorter` document against optional base reducer state.
-pub fn compile_document(
+fn compile_document_inner(
base: &ReducerState,
room: &str,
text: &str,
+ ingest_id: Option<String>,
+ ingest_line: Option<usize>,
) -> Result<CompileResult, CompileError> {
let room_key = room.trim();
let scope = scope_from_room_wire(room_key);
let validated = validate_ingest_document(base, text, &scope).map_err(|(_, message, hint)| {
+ let parse_error = dsl::parse_full(text).err().map(|e| e.to_string());
CompileError {
ok: false,
error: message,
hint,
+ parse_error,
}
})?;
@@ -200,15 +236,27 @@ pub fn compile_document(
threads: threads_in_document(text),
rankings: rankings_for_simulated(&simulated, &scope, room_key, &validated.doc),
stats: document_stats(&validated.doc),
+ ingest_id,
+ ingest_line,
})
}
+/// Validate and simulate one `.sorter` document against optional base reducer state.
+pub fn compile_document(
+ base: &ReducerState,
+ room: &str,
+ text: &str,
+) -> Result<CompileResult, CompileError> {
+ compile_document_inner(base, room, text, None, None)
+}
+
fn ingest_parse_error(raw: &str) -> Option<String> {
dsl::parse_full(raw).err().map(|e| e.to_string())
}
-fn load_events_from_jsonl(path: &Path) -> Result<(Vec<(usize, Event)>, Vec<BadJsonLine>), std::io::Error> {
+fn load_events_from_jsonl(path: &Path) -> Result<(usize, Vec<(usize, Event)>, Vec<BadJsonLine>), std::io::Error> {
let text = std::fs::read_to_string(path)?;
+ let total_lines = text.lines().count();
let mut events = Vec::new();
let mut bad_json_lines = Vec::new();
for (idx, line) in text.lines().enumerate() {
@@ -225,61 +273,90 @@ fn load_events_from_jsonl(path: &Path) -> Result<(Vec<(usize, Event)>, Vec<BadJs
}),
}
}
- Ok((events, bad_json_lines))
+ Ok((total_lines, events, bad_json_lines))
}
-/// Replay a JSONL event log into reducer state (same rules as server boot).
-pub fn load_reducer_from_jsonl(path: &Path) -> Result<(ReducerState, Vec<BadJsonLine>), std::io::Error> {
- let (events, bad_json_lines) = load_events_from_jsonl(path)?;
+fn replay_events(events: &[(usize, Event)]) -> ReducerState {
let mut state = ReducerState::default();
for (_line_no, ev) in events {
- state.apply_event(ev);
+ state.apply_event(ev.clone());
}
- Ok((state, bad_json_lines))
+ state
}
-/// Scan an events.jsonl for corrupt JSON lines and ingests that fail DSL replay.
-pub fn scan_jsonl(path: &Path) -> Result<ScanResult, std::io::Error> {
- let text = std::fs::read_to_string(path)?;
- let total_lines = text.lines().count();
- let (events, bad_json_lines) = load_events_from_jsonl(path)?;
+/// Replay a JSONL event log into reducer state (same rules as server boot).
+pub fn load_reducer_from_jsonl(path: &Path) -> Result<(ReducerState, Vec<BadJsonLine>), std::io::Error> {
+ let (_total_lines, events, bad_json_lines) = load_events_from_jsonl(path)?;
+ Ok((replay_events(&events), bad_json_lines))
+}
- let mut malformed_ingests = Vec::new();
- let mut skipped_ingests = 0usize;
- let mut state = ReducerState::default();
- let parsed_events = events.len();
+/// Find one ingest in a log and compile it against all prior events as base state.
+pub fn compile_ingest_from_log(path: &Path, ingest_id: &str) -> Result<CompileResult, CompileIngestError> {
+ let (_total_lines, events, bad_json_lines) = load_events_from_jsonl(path).map_err(CompileIngestError::Io)?;
+ if !bad_json_lines.is_empty() {
+ return Err(CompileIngestError::Compile(CompileError {
+ ok: false,
+ error: format!("jsonl has {} corrupt line(s)", bad_json_lines.len()),
+ hint: Some("fix the log or use `sorterc scan`".into()),
+ parse_error: None,
+ }));
+ }
+
+ let needle = ingest_id.trim();
+ let mut found: Option<(usize, Ingest)> = None;
+ let mut prior: Vec<(usize, Event)> = Vec::new();
for (line_no, ev) in events {
if let Event::Ingest(ref ing) = ev {
- if let Some(reason) = ingest_parse_error(&ing.raw) {
+ if ing.id == needle {
+ found = Some((line_no, ing.clone()));
+ break;
+ }
+ }
+ prior.push((line_no, ev));
+ }
+
+ let (line_no, ing) = found.ok_or_else(|| CompileIngestError::NotFound(needle.to_string()))?;
+ let base = replay_events(&prior);
+ compile_document_inner(&base, &ing.room_id, &ing.raw, Some(ing.id.clone()), Some(line_no))
+ .map_err(CompileIngestError::Compile)
+}
+
+/// Scan an events.jsonl for corrupt JSON lines and ingests whose DSL fails to parse.
+///
+/// This does not replay the log (which would run rank centrality on every ingest and
+/// can take minutes on real logs). It matches what the server skips on boot: parse failure.
+pub fn scan_jsonl(path: &Path) -> Result<ScanResult, std::io::Error> {
+ let (total_lines, events, bad_json_lines) = load_events_from_jsonl(path)?;
+
+ let mut malformed_ingests = Vec::new();
+ let mut ingest_events = 0usize;
+
+ for (line_no, ev) in &events {
+ if let Event::Ingest(ing) = ev {
+ ingest_events += 1;
+ if let Some(parse_error) = ingest_parse_error(&ing.raw) {
malformed_ingests.push(MalformedIngest {
- line: line_no,
+ line: *line_no,
id: ing.id.clone(),
room_id: ing.room_id.clone(),
thread_tag: ing.thread_tag.clone(),
- reason,
+ parse_error,
});
}
- let before = state.ingests_by_id.len();
- state.apply_event(ev);
- if state.ingests_by_id.len() == before {
- skipped_ingests += 1;
- }
- } else {
- state.apply_event(ev);
}
}
- let ok = bad_json_lines.is_empty() && malformed_ingests.is_empty() && skipped_ingests == 0;
+ let ok = bad_json_lines.is_empty() && malformed_ingests.is_empty();
Ok(ScanResult {
ok,
path: path.display().to_string(),
total_lines,
- parsed_events,
+ parsed_events: events.len(),
+ ingest_events,
bad_json_lines,
malformed_ingests,
- skipped_ingests,
})
}
@@ -330,4 +407,25 @@ mod tests {
assert!(!report.ok);
assert_eq!(report.bad_json_lines.len(), 1);
}
+
+ #[test]
+ fn scan_reports_dsl_parse_error_detail() {
+ let dir = tempfile::tempdir().unwrap();
+ let path = dir.path().join("events.jsonl");
+ let ingest = serde_json::json!({
+ "type": "ingest",
+ "ts": 1,
+ "id": "bad-ingest-id",
+ "raw": "{ no closing brace\n~/a { body }\n~/a 1:0 ~/b",
+ "principal": "test",
+ "room_id": "public",
+ "thread_tag": "t",
+ });
+ std::fs::write(&path, format!("{ingest}\n")).unwrap();
+ let report = scan_jsonl(&path).unwrap();
+ assert!(!report.ok);
+ assert_eq!(report.malformed_ingests.len(), 1);
+ assert_eq!(report.malformed_ingests[0].id, "bad-ingest-id");
+ assert!(report.malformed_ingests[0].parse_error.contains("parse error"));
+ }
}
diff --git a/sorterc/readme.md b/sorterc/readme.md
index 1ebcc3fc935541ea9e47e0458ec67a750fe19fff..e388615c34b40914ca51fc7a79cb738eebc2909b 100644
--- a/sorterc/readme.md
+++ b/sorterc/readme.md
@@ -60,21 +60,33 @@ Rankings use the same structure as the server's dry-run check: parent scope, con
### `scan` — lint an `events.jsonl`
-Reads a JSONL event log and reports problems without starting a server.
+Fast single-pass check. Does **not** replay the log (replay runs rank centrality on every ingest and gets slow fast).
```bash
cargo run -p sorterc -- scan events.jsonl
-cargo run -p sorterc -- scan events.jsonl --pretty
+cargo run -p sorterc -- scan events.jsonl --json
+cargo run -p sorterc -- scan events.jsonl --json --pretty
```
+Default output is human-readable with a blank line between each problem. Use `--json` for machine output.
+
Reports:
- **bad JSON lines** — lines that are not valid JSON
-- **malformed ingests** — ingest events whose `raw` DSL fails to parse
-- **skipped ingests** — ingests dropped during replay (same behavior as server boot)
+- **malformed ingests** — ingest events whose `raw` DSL fails to parse, with full `parse_error` text
Exits 0 when clean, 1 when any issue is found.
+### `compile --ingest` — compile one event from a log
+
+Replay all events **before** the target ingest as base state, then compile that ingest's DSL:
+
+```bash
+cargo run -p sorterc -- compile --ingest cabd8adc-57ae-402d-a940-8e24339ac451 --from events.jsonl --pretty
+```
+
+Output includes `ingest_id`, `ingest_line`, and rankings for that post only. This may take a while for ingests late in a large log (full replay up to that point).
+
## Typical uses
- Iterate on `.sorter` files in an editor and pipe through `compile` to see rankings instantly
diff --git a/sorterc/src/main.rs b/sorterc/src/main.rs
index 71382c180085cb0ad71043c852f8db5d3a48a284..5687c850d16973d28749380b8dc89f3bcebbfc56 100644
--- a/sorterc/src/main.rs
+++ b/sorterc/src/main.rs
@@ -22,9 +22,15 @@ struct Cli {
enum Command {
/// Parse and simulate a .sorter document; emit ranking JSON to stdout.
Compile {
- /// `.sorter` file, or `-` f
… preview truncated; 6,030 characters omittedHardlinks — judgments / attempts / prompt
judgments
attempts
Prompt text is loaded only by the download route.