A replaces fragile string-splitting canonicalization with a properly designed, composable url_rules engine (real URL parsing via the `url` crate, per-domain rules, alias detection, YouTube/Reddit normalization) backed by thorough new unit tests and documentation — a focused, correct, lasting design improvement. B is a much larger but self-described 'first pass' refactor that mostly relocates existing REST-handler logic into one giant monolithic rpc.rs match statement, trading modularity for a batched RPC surface without clearly improving code quality, making its long-term value less certain despite the larger diff.
constitution · epochs · watch · epoch 3
comparison
c_77729db919ab (tommy-mor) vs c_2595b6007624 (tommy-mor)
download prompt · raw event · cmp_02aac909c8ad0f
council reasoning
B delivers a lasting domain/API redesign: permission boundaries become rooms (RoomCreated, ScopeId::Room, room_id on Ingest) separate from forum thread_tag, with a unified batch RPC surface that replaces many REST handlers and scopes the CLI (public/private). A is a solid identity rewrite (full https ItemIds plus a composable url_rules engine and breadcrumb/parent fixes), but a large share of that patch is mechanical ID/string updates across tests versus B’s structural event/reducer/API changes.
Side A introduces a reusable URL canonicalization subsystem (`server/src/url_rules/`), converts `ItemId` to canonical full URLs, centralizes parsing/parent/breadcrumb logic, and updates projection and Reddit handling to consistently normalize persisted IDs. Side B is a sweeping API refactor that replaces many endpoints with a single RPC layer and room-scoped commands, but much of the patch is code movement and interface consolidation rather than new core behavior, making A's normalization design and identity fixes more foundational and durable.
sides
A — c_77729db919ab (tommy-mor)
message
[239c074b] url schema stuff
diff preview
diff --git a/AGENTS.md b/AGENTS.md
index 426a88e7c1da54fe0a28c5c76fa4e1f1bc117fcf..e60b9ba6012593361ef10e8fdd9439cd9932e09b 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -58,3 +58,4 @@ Use **tmux** for `cargo run --package sorter2-server` (dev server). Rebuild afte
- First `cargo test` / `cargo build --release` is slow; Clojure smoke test always does a release build.
- `legacy/` and `ideas/` are not part of the workspace build.
+- **ItemId** for web URLs is a canonical full URL (`https://reddit.com/r/rust`). Rules live in [`server/src/url_rules/`](server/src/url_rules/) (composable Rust, not a config DSL). After changing canonicalization rules, rebuild the projection: `cargo run --package sorter2-server -- replay-index`.
diff --git a/Cargo.lock b/Cargo.lock
index 0dd4fce5fb6400ae153cca4e3dbf5a5158e6d8b4..49a908ef935c430dbe63c6a28d8a24e38b489486 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -1951,6 +1951,7 @@ dependencies = [
"tower-http 0.5.2",
"tracing",
"tracing-subscriber",
+ "url",
"urlencoding",
]
diff --git a/REPLAY.sh b/REPLAY.sh
new file mode 100755
index 0000000000000000000000000000000000000000..f2dbd8aea60c02d2feef74805f7ef5c2b7022537
--- /dev/null
+++ b/REPLAY.sh
@@ -0,0 +1,2 @@
+cargo run --package sorter2-server -- replay-index
+
diff --git a/server/Cargo.toml b/server/Cargo.toml
index 27f552c20b97ef28cdde4cb6b1a4980375135111..ad4912791aff59fb1d3293f66ad381ae618cd60b 100644
--- a/server/Cargo.toml
+++ b/server/Cargo.toml
@@ -24,6 +24,7 @@ async-stream = "0.3"
futures-util = { version = "0.3", default-features = false, features = ["std"] }
rand = "0.8"
urlencoding = "2"
+url = "2"
durable = { path = "../durable" }
[dev-dependencies]
diff --git a/server/src/entity_store.rs b/server/src/entity_store.rs
index d5d17c3676e4a8ddec998e9f5a9dbafe9c2d9d0e..d29f39aecca6f12cdcf263cf77c3654eb4ee6cfa 100644
--- a/server/src/entity_store.rs
+++ b/server/src/entity_store.rs
@@ -124,7 +124,7 @@ mod tests {
fn round_trip_payload() {
let tmp = tempfile::tempdir().unwrap();
let store = EntityStore::open(tmp.path()).unwrap();
- let id = ItemId::parse("reddit.com/r/rust").unwrap();
+ let id = ItemId::from_url("https://reddit.com/r/rust").unwrap();
let payload = json!({"kind": "t5", "data": {"display_name": "rust"}});
store.put(&id, &payload).unwrap();
diff --git a/server/src/event_log.rs b/server/src/event_log.rs
index 36f5b406084065b608735987cdb483c236e03081..2c9290b6fdbf2c2ad1c0f1ffd7374b2d9cc97f36 100644
--- a/server/src/event_log.rs
+++ b/server/src/event_log.rs
@@ -199,7 +199,7 @@ mod tests {
log.append(&sample_record(
1,
Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
},
))
.await
@@ -237,7 +237,7 @@ mod tests {
let path = tmp.path().join("events.jsonl");
let log = EventLog::new(&path);
let event = Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
};
log.append(&sample_record(1, event)).await.unwrap();
@@ -255,7 +255,7 @@ mod tests {
let path = tmp.path().join("events.jsonl");
std::fs::write(
&path,
- r#"{"type":"node_ensured","id":"reddit.com/r/rust"}
+ r#"{"type":"node_ensured","id":"https://reddit.com/r/rust"}
{"schema":1,"seq":1,"ts":1,"event":{"type":"vote_recorded","ts":1,"a":"a","b":"b","ratio_left":2,"ratio_right":1,"scope":""}}
"#,
)
@@ -295,7 +295,7 @@ mod tests {
log.append(&sample_record(
1,
Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
},
))
.await
@@ -303,7 +303,7 @@ mod tests {
log.append(&sample_record(
3,
Event::NodeEnsured {
- id: "reddit.com/r/python".into(),
+ id: "https://reddit.com/r/python".into(),
},
))
.await
diff --git a/server/src/journal.rs b/server/src/journal.rs
index 521a108019de1ea870d14c4fafbfe572c20ce0de..50bc89f976edb82b7b0e49e954a8eccbbe82bf87 100644
--- a/server/src/journal.rs
+++ b/server/src/journal.rs
@@ -141,10 +141,10 @@ mod tests {
let j2 = journal.clone();
let (r1, r2) = tokio::join!(
j1.append(Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
}),
j2.append(Event::NodeEnsured {
- id: "reddit.com/r/python".into(),
+ id: "https://reddit.com/r/python".into(),
}),
);
r1.unwrap();
@@ -153,10 +153,10 @@ mod tests {
assert_eq!(projection_store.last_applied_event_count().unwrap(), 2);
let tree = projection_store.load_tree().unwrap();
assert!(tree
- .get(&ItemId::parse("reddit.com/r/rust").unwrap())
+ .get(&ItemId::parse("https://reddit.com/r/rust").unwrap())
.is_some());
assert!(tree
- .get(&ItemId::parse("reddit.com/r/python").unwrap())
+ .get(&ItemId::parse("https://reddit.com/r/python").unwrap())
.is_some());
}
@@ -170,7 +170,7 @@ mod tests {
1,
1,
Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
},
))
.await
@@ -186,7 +186,7 @@ mod tests {
1,
1,
Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
},
)],
)
@@ -202,7 +202,7 @@ mod tests {
);
journal
.append(Event::NodeEnsured {
- id: "reddit.com/r/python".into(),
+ id: "https://reddit.com/r/python".into(),
})
.await
.unwrap();
@@ -227,13 +227,13 @@ mod tests {
journal
.append_many(vec![
Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
},
Event::NodeEnsured {
- id: "reddit.com/r/python".into(),
+ id: "https://reddit.com/r/python".into(),
},
Event::NodeEnsured {
- id: "reddit.com/r/clojure".into(),
+ id: "https://reddit.com/r/clojure".into(),
},
])
.await
@@ -245,7 +245,7 @@ mod tests {
assert_eq!(projection_store.last_applied_event_count().unwrap(), 3);
let tree = projection_store.load_tree().unwrap();
assert!(tree
- .get(&ItemId::parse("reddit.com/r/clojure").unwrap())
+ .get(&ItemId::parse("https://reddit.com/r/clojure").unwrap())
.is_some());
}
}
diff --git a/server/src/lib.rs b/server/src/lib.rs
index 9bd5f76fd1406b9b1be4c272f4ba8647edde2678..5c02c8e704e4664453bad75d819df8a067668176 100644
--- a/server/src/lib.rs
+++ b/server/src/lib.rs
@@ -9,6 +9,7 @@ pub mod journal;
pub mod pair;
pub mod parser;
pub mod path_types;
+pub mod url_rules;
pub mod projection_apply;
pub mod projection_store;
pub mod ranking;
diff --git a/server/src/pair.rs b/server/src/pair.rs
index 43f780ba6ea6ce1cdc2e1f4cbb252ba8a10684b9..815a97b80e3e9f348e0937a4f147f2862018edb0 100644
--- a/server/src/pair.rs
+++ b/server/src/pair.rs
@@ -381,42 +381,42 @@ mod tests {
#[test]
fn suggest_prefers_unvoted_pair() {
- let parent = ItemId::parse("reddit.com/r/rust").unwrap();
+ let parent = ItemId::parse("https://reddit.com/r/rust").unwrap();
let mut tree = seed_children(
&parent,
&[
- "reddit.com/r/rust/a",
- "reddit.com/r/rust/b",
- "reddit.com/r/rust/c",
+ "https://reddit.com/r/rust/a",
+ "https://reddit.com/r/rust/b",
+ "https://reddit.com/r/rust/c",
],
);
let vote =
- VoteData::from_recorded(1, "reddit.com/r/rust/a", "reddit.com/r/rust/b", 2, 1).unwrap();
+ VoteData::from_recorded(1, "https://reddit.com/r/rust/a", "https://reddit.com/r/rust/b", 2, 1).unwrap();
tree.apply_vote(&parent, vote);
let group = tree.get(&parent).unwrap().local_ranking.clone();
let pool = children_of(&tree, &parent);
let (l, r) = suggest_next_pair_in_pool(&group, &pool, None).unwrap();
- let voted_ab = (l.as_str() == "reddit.com/r/rust/a" && r.as_str() == "reddit.com/r/rust/b")
- || (l.as_str() == "reddit.com/r/rust/b" && r.as_str() == "reddit.com/r/rust/a");
+ let voted_ab = (l.as_str() == "https://reddit.com/r/rust/a" && r.as_str() == "https://reddit.com/r/rust/b")
+ || (l.as_str() == "https://reddit.com/r/rust/b" && r.as_str() == "https://reddit.com/r/rust/a");
assert!(!voted_ab);
}
#[test]
fn suggest_bridges_separate_components() {
- let parent = ItemId::parse("reddit.com/r/rust").unwrap();
+ let parent = ItemId::parse("https://reddit.com/r/rust").unwrap();
let mut tree = seed_children(
&parent,
&[
- "reddit.com/r/rust/a",
- "reddit.com/r/rust/b",
- "reddit.com/r/rust/c",
- "reddit.com/r/rust/d",
+ "https://reddit.com/r/rust/a",
+ "https://reddit.com/r/rust/b",
+ "https://reddit.com/r/rust/c",
+ "https://reddit.com/r/rust/d",
],
);
let ab =
- VoteData::from_recorded(1, "reddit.com/r/rust/a", "reddit.com/r/rust/b", 2, 1).unwrap();
+ VoteData::from_recorded(1, "https://reddit.com/r/rust/a", "https://reddit.com/r/rust/b", 2, 1).unwrap();
let cd =
- VoteData::from_recorded(2, "reddit.com/r/rust/c", "reddit.com/r/rust/d", 2, 1).unwrap();
+ VoteData::from_recorded(2, "https://reddit.com/r/rust/c", "https://reddit.com/r/rust/d", 2, 1).unwrap();
tree.apply_vote(&parent, ab);
tree.apply_vote(&parent, cd);
let group = tree.get(&parent).unwrap().local_ranking.clone();
@@ -424,37 +424,37 @@ mod tests {
let pair = suggest_next_pair_in_pool(&group, &pool, None).unwrap();
let chosen = pair_set(&pair);
let from_ab =
- chosen.contains("reddit.com/r/rust/a") || chosen.contains("reddit.com/r/rust/b");
+ chosen.contains("https://reddit.com/r/rust/a") || chosen.contains("https://reddit.com/r/rust/b");
let from_cd =
- chosen.contains("reddit.com/r/rust/c") || chosen.contains("reddit.com/r/rust/d");
+ chosen.contains("https://reddit.com/r/rust/c") || chosen.contains("https://reddit.com/r/rust/d");
assert!(from_ab && from_cd, "expected bridge pair, got {:?}", chosen);
}
#[test]
fn suggest_prefers_attach_over_isolate_pair_among_many_unranked() {
- let parent = ItemId::parse("reddit.com/r/rust").unwrap();
+ let parent = ItemId::parse("https://reddit.com/r/rust").unwrap();
let mut tree = seed_children(
&parent,
&[
- "reddit.com/r/rust/a",
- "reddit.com/r/rust/b",
- "reddit.com/r/rust/c",
- "reddit.com/r/rust/d",
- "reddit.com/r/rust/e",
+ "https://reddit.com/r/rust/a",
+ "https://reddit.com/r/rust/b",
+ "https://reddit.com/r/rust/c",
+ "https://reddit.com/r/
… preview truncated; 51,799 characters omittedB — c_2595b6007624 (tommy-mor)
message
[96b6da05] rpc + reducer changes first pass
diff preview
diff --git a/cli/src/main.rs b/cli/src/main.rs
index 630c5dea1f78c0ec9bc53e6b96234a0dc75bb705..8d0442959f4332bafe499a2a8cdf364731a06871 100644
--- a/cli/src/main.rs
+++ b/cli/src/main.rs
@@ -21,8 +21,9 @@ struct Cli {
cmd: Option<Command>,
}
+/// Commands scoped to a room (`public` or `shortid/slug`).
#[derive(Subcommand, Debug)]
-enum Command {
+enum ScopedCmd {
/// Browse the garden (ontology) — light mode, ranked by votes
Garden {
#[command(subcommand)]
@@ -174,6 +175,23 @@ enum Command {
#[arg(long)]
json: bool,
},
+}
+
+#[derive(Subcommand, Debug)]
+enum Command {
+ /// Public site (same as room `public`)
+ Public {
+ #[command(subcommand)]
+ sub: ScopedCmd,
+ },
+ /// Private room id (`shortid/slug` from `room create`)
+ Private {
+ /// Room id, e.g. `a1b2c3d/my-project`
+ #[arg(value_name = "ROOM_ID")]
+ room: String,
+ #[command(subcommand)]
+ sub: ScopedCmd,
+ },
/// Show all activity since you last posted (global feed)
///
@@ -628,6 +646,37 @@ fn http_client() -> Result<reqwest::Client> {
.build()?)
}
+async fn send_rpc(
+ client: &reqwest::Client,
+ base: &str,
+ bearer: Option<&str>,
+ commands: Vec<RpcCommand>,
+) -> Result<RpcBatchResponse> {
+ let url = format!("{}/api/v0/rpc", base.trim_end_matches('/'));
+ let mut req = client.post(url).json(&RpcBatch(commands));
+ if let Some(b) = bearer {
+ req = req.header("Authorization", format!("Bearer {}", b));
+ }
+ let resp = req.send().await?;
+ let status = resp.status();
+ let text = resp.text().await.unwrap_or_default();
+ if !status.is_success() {
+ return Err(anyhow!("rpc HTTP {}: {}", status, text.trim()));
+ }
+ serde_json::from_str(&text).map_err(|e| anyhow!("rpc response: {e}"))
+}
+
+fn rpc_line_ok(line: &RpcLine) -> Result<&RpcResult> {
+ if !line.ok {
+ let mut m = line.error.clone().unwrap_or_else(|| "rpc error".into());
+ if let Some(h) = &line.hint {
+ m.push_str(&format!("\nhint: {h}"));
+ }
+ return Err(anyhow!(m));
+ }
+ line.result.as_ref().ok_or_else(|| anyhow!("rpc missing result"))
+}
+
/// Normalize ontology path for API. Accepts path with or without ~/ (shell expands ~ to $HOME).
/// Returns a bare slug path (e.g. `languages/python`) with no leading `/` or `~/`.
/// Call `ontology_path_for_api_query` before sending `item=` / `parent=` params so the server
@@ -729,231 +778,262 @@ fn write_secret_file(name: &str, contents: &str) -> Result<()> {
Ok(())
}
-#[tokio::main]
-async fn main() -> Result<()> {
- let Cli { cmd, server } = Cli::parse();
-
- // If no command provided, print the guide
- let Some(cmd) = cmd else {
- print!("{}", include_str!("../GUIDE.sorter"));
- return Ok(());
- };
-
- let base = server.trim_end_matches('/');
-
- match cmd {
- Command::Healthz { json } => {
- let client = http_client()?;
- let url = format!("{base}/healthz");
- let body = client.get(url).send().await?.text().await?;
- if json {
- // Wrap plain text response in a JSON object
- println!("{}", serde_json::json!({ "ok": true, "body": body.trim() }));
- } else {
- println!("{body}");
- }
- }
-
- Command::Search { query, json } => {
- let client = http_client()?;
- let url = format!("{base}/api/v0/search?q={}", urlencoding::encode(&query));
- let resp: slug_types::SearchResponse = expect_json(client.get(url).send().await?).await?;
- if json {
- println!("{}", serde_json::to_string_pretty(&resp)?);
- } else {
- if !resp.items.is_empty() {
- println!("items ({})", resp.items.len());
- for item in &resp.items {
- print!(" {}", item.path);
- if let Some(body) = &item.body {
- let first_line = body.lines().next().unwrap_or("").trim();
- if !first_line.is_empty() {
- print!(" {}", first_line);
+async fn run_scoped(base: &str, room: &str, sub: ScopedCmd) -> Result<()> {
+ let room = room.trim();
+ let client = http_client()?;
+ match sub {
+ ScopedCmd::Garden { sub } => match sub {
+ GardenCmd::Tree { json } => {
+ let batch = send_rpc(&client, base, None, vec![RpcCommand::GetLeaves { room: room.to_string() }]).await?;
+ match rpc_line_ok(&batch.results[0])? {
+ RpcResult::Leaves(resp) => {
+ if json {
+ println!("{}", serde_json::to_string_pretty(&resp)?);
+ } else {
+ for p in &resp.paths {
+ println!("~/{}", p);
}
}
- println!();
- }
- }
- if !resp.threads.is_empty() {
- if !resp.items.is_empty() { println!(); }
- println!("threads ({})", resp.threads.len());
- let now_ms = std::time::SystemTime::now()
- .duration_since(std::time::UNIX_EPOCH)
- .unwrap_or_default()
- .as_millis() as i64;
- for t in &resp.threads {
- println!(" {} {}n {}", t.tag, t.post_count, slug_types::timeago::timeago(now_ms, t.last_activity));
- }
- }
- if !resp.posts.is_empty() {
- if !resp.items.is_empty() || !resp.threads.is_empty() { println!(); }
- println!("posts ({})", resp.posts.len());
- let now_ms = std::time::SystemTime::now()
- .duration_since(std::time::UNIX_EPOCH)
- .unwrap_or_default()
- .as_millis() as i64;
- for p in &resp.posts {
- let first_line = p.snippet.lines().next().unwrap_or("").trim();
- println!(" {} · {} {}", p.thread, slug_types::timeago::timeago(now_ms, p.ts), first_line);
- }
- }
- if resp.items.is_empty() && resp.threads.is_empty() && resp.posts.is_empty() {
- println!("no results");
- }
- }
- }
-
- Command::Garden { sub } => match sub {
- GardenCmd::Tree { json } => {
- let client = http_client()?;
- let url = format!("{base}/api/v0/leaves");
- let builder = client.get(url);
- let resp: LeavesResponse = expect_json(builder.send().await?).await?;
- if json {
- println!("{}", serde_json::to_string_pretty(&resp)?);
- } else {
- for p in &resp.paths {
- println!("~/{}", p);
}
+ _ => return Err(anyhow!("unexpected RPC result")),
}
}
-
GardenCmd::Body { path, json, full } => {
let path = normalize_ontology_path_input(&path).map_err(anyhow::Error::msg)?;
let item_q = ontology_path_for_api_query(&path);
- let client = http_client()?;
- let mut url = format!("{base}/api/v0/item?item={}", urlencoding::encode(&item_q));
- if full {
- url.push_str("&full=true");
- }
- let builder = client.get(url);
- let resp: ItemResponse = expect_json(builder.send().await?).await?;
- if json {
- println!("{}", serde_json::to_string_pretty(&resp)?);
- } else {
- print_item_response(&resp);
+ let batch = send_rpc(
+ &client,
+ base,
+ None,
+ vec![RpcCommand::GetGardenItem {
+ room: room.to_string(),
+ item_path: item_q,
+ full: Some(full),
+ }],
+ )
+ .await?;
+ match rpc_line_ok(&batch.results[0])? {
+ RpcResult::GardenItem(resp) => {
+ if json {
+ println!("{}", serde_json::to_string_pretty(&resp)?);
+ } else {
+ print_item_response(&resp);
+ }
+ }
+ _ => return Err(anyhow!("unexpected RPC result")),
}
}
-
GardenCmd::Children { paths, depth, json } => {
let paths: Vec<String> = paths
.iter()
.map(|p| normalize_ontology_path_input(p).map_err(anyhow::Error::msg))
.collect::<Result<Vec<_>>>()?;
- let client = http_client()?;
let parent_param = paths
.iter()
.map(|p| ontology_path_for_api_query(p))
.collect::<Vec<_>>()
.join(",");
- let mut url = format!("{base}/api/v0/rank?parent={}", urlencoding::encode(&parent_param));
- if let Some(d) = depth {
- url.push_str(&format!("&depth={d}"));
- }
- let builder = client.get(url);
- let resp: RankResponse = expect_json(builder.send().await?).await?;
-
- if json {
- println!("{}", serde_json::to_string_pretty(&resp)?);
- } else {
- print_rank_response(&resp);
+ let batch = send_rpc(
+ &client,
+ base,
+ None,
+ vec![RpcCommand::GetGardenRank {
+ room: room.to_string(),
+ parent_path: parent_param,
+ depth,
+ offset: None,
+ limit: None,
+ percent: None,
+ }],
+ )
+ .await?;
+ match rpc_line_ok(&batch.results[0])? {
+ RpcResult::GardenRank(resp) => {
+ if json {
+ println!("{}", serde_json::to_string_pretty(&resp)?);
+ } else {
+ print_rank_response(&resp);
+ }
+ }
+ _ => return Err(anyhow!("unexpected RPC result")),
}
}
-
GardenCmd::Pair { path, json } => {
let path = normalize_ontology_path_input(&path).map_err(anyhow::Error::msg)?;
let parent_q = ontology_path_for_api_query(&path);
- let client = http_client()?;
- let url = format!("{base}/api/v0/pair?parent={}", urlencoding::encode(&parent_q));
- let builder = client.get(url);
- let resp: PairResponse = expect_json(builder.send().await?).await?;
- if json {
- println!("{}", serde_json::to_string_pretty(&resp)?);
- } else {
- print_pair_response(&resp);
+ let batch = send_rpc(
+ &client,
+ base,
+ None,
+ vec![RpcCommand::GetPair {
+ room: room.to_string(),
+ parent_path: parent_q,
+ }]
… preview truncated; 237,234 characters omittedHardlinks — judgments / attempts / prompt
judgments
attempts
Prompt text is loaded only by the download route.