Side B is a substantial, well-tested architectural rework: it introduces a composable url_rules engine/registry, migrates ItemId to canonical full URLs, fixes breadcrumb/parent logic for structural path segments, and updates all call sites and tests consistently, providing lasting infrastructure value. Side A is a small, useful bugfix (skip pinned Reddit posts) with a good test, but it is narrow in scope compared to B's broad canonicalization redesign.
constitution · epochs · watch · epoch 3
comparison
c_bc8c17a00ed7 (tommy-mor) vs c_77729db919ab (tommy-mor)
download prompt · raw event · cmp_d2d558ad7bec22
council reasoning
B replaces ad-hoc schemeless path IDs with a real full-URL ItemId model and a composable url_rules engine (Reddit/YouTube canonicalization, breadcrumbs, parent hierarchy), which is core lasting design wired through the stack. A is a correct, tested import filter for stickied/pinned posts, but it is a narrow local improvement versus B’s foundational identity/schema work.
Side B introduces a substantial URL canonicalization architecture by extracting normalization into a new `url_rules` module, converting `ItemId` to canonical HTTPS URLs, updating parsing, parent/breadcrumb logic, projection application, and Reddit mapping, with broad test coverage. Side A is a useful targeted bug fix that skips stickied/pinned Reddit posts during subreddit import and adds a regression test, but its impact is much narrower than the lasting infrastructure and consistency improvements in Side B.
sides
A — c_bc8c17a00ed7 (tommy-mor)
message
[03cd8f2e] Skip pinned Reddit posts when importing subreddit listings. Co-authored-by: Cursor <cursoragent@cursor.com>
diff preview
diff --git a/server/src/reddit.rs b/server/src/reddit.rs
index f409764c1e1f36216f1b08107043c2eab905694c..fa5577f8ee0a8c53ff4dbec988a90a5d2fc3cdee 100644
--- a/server/src/reddit.rs
+++ b/server/src/reddit.rs
@@ -661,6 +661,7 @@ pub fn map_children_url(id: &ItemId, api_base: &str) -> String {
/// Parse a subreddit listing payload into `(child_id, child_payload)` entries.
/// Each child id is the post's permalink under `reddit.com/…`, and the payload
/// is the raw `{kind, data}` listing element (persisted per child).
+/// Pinned / stickied posts are skipped.
fn parse_children(_parent: &ItemId, payload: &Value) -> Vec<(ItemId, Value)> {
let mut out = Vec::new();
let children = match payload.pointer("/data/children").and_then(|c| c.as_array()) {
@@ -668,6 +669,9 @@ fn parse_children(_parent: &ItemId, payload: &Value) -> Vec<(ItemId, Value)> {
None => return out,
};
for child in children {
+ if child_is_pinned(child) {
+ continue;
+ }
let permalink = match child.pointer("/data/permalink").and_then(|p| p.as_str()) {
Some(p) if !p.is_empty() => p,
_ => continue,
@@ -680,6 +684,15 @@ fn parse_children(_parent: &ItemId, payload: &Value) -> Vec<(ItemId, Value)> {
out
}
+fn child_is_pinned(child: &Value) -> bool {
+ let data = match child.get("data") {
+ Some(d) => d,
+ None => return false,
+ };
+ data.get("stickied").and_then(|v| v.as_bool()) == Some(true)
+ || data.get("pinned").and_then(|v| v.as_bool()) == Some(true)
+}
+
fn parse_reddit_view(id: &ItemId, v: &Value) -> Option<crate::reducer::EntityData> {
let segments: Vec<&str> = id.as_str().split('/').collect();
@@ -853,4 +866,46 @@ mod tests {
Some("http://v3.redgifs.com/watch/impossibleprestigioushedgehog")
);
}
+
+ #[test]
+ fn parse_children_skips_pinned_posts() {
+ let payload = serde_json::json!({
+ "kind": "Listing",
+ "data": {
+ "children": [
+ {
+ "kind": "t3",
+ "data": {
+ "title": "Official rules (pinned)",
+ "permalink": "/r/rust/comments/pin/official_rules/",
+ "stickied": true
+ }
+ },
+ {
+ "kind": "t3",
+ "data": {
+ "title": "Also pinned via pinned field",
+ "permalink": "/r/rust/comments/pin2/also_pinned/",
+ "pinned": true
+ }
+ },
+ {
+ "kind": "t3",
+ "data": {
+ "title": "Normal post",
+ "permalink": "/r/rust/comments/aaa/normal_post/",
+ "stickied": false
+ }
+ }
+ ]
+ }
+ });
+ let parent = ItemId::from_url("https://reddit.com/r/rust").unwrap();
+ let children = parse_children(&parent, &payload);
+ assert_eq!(children.len(), 1);
+ assert_eq!(
+ children[0].0.as_str(),
+ "https://reddit.com/r/rust/comments/aaa"
+ );
+ }
}
B — c_77729db919ab (tommy-mor)
message
[239c074b] url schema stuff
diff preview
diff --git a/AGENTS.md b/AGENTS.md
index 426a88e7c1da54fe0a28c5c76fa4e1f1bc117fcf..e60b9ba6012593361ef10e8fdd9439cd9932e09b 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -58,3 +58,4 @@ Use **tmux** for `cargo run --package sorter2-server` (dev server). Rebuild afte
- First `cargo test` / `cargo build --release` is slow; Clojure smoke test always does a release build.
- `legacy/` and `ideas/` are not part of the workspace build.
+- **ItemId** for web URLs is a canonical full URL (`https://reddit.com/r/rust`). Rules live in [`server/src/url_rules/`](server/src/url_rules/) (composable Rust, not a config DSL). After changing canonicalization rules, rebuild the projection: `cargo run --package sorter2-server -- replay-index`.
diff --git a/Cargo.lock b/Cargo.lock
index 0dd4fce5fb6400ae153cca4e3dbf5a5158e6d8b4..49a908ef935c430dbe63c6a28d8a24e38b489486 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -1951,6 +1951,7 @@ dependencies = [
"tower-http 0.5.2",
"tracing",
"tracing-subscriber",
+ "url",
"urlencoding",
]
diff --git a/REPLAY.sh b/REPLAY.sh
new file mode 100755
index 0000000000000000000000000000000000000000..f2dbd8aea60c02d2feef74805f7ef5c2b7022537
--- /dev/null
+++ b/REPLAY.sh
@@ -0,0 +1,2 @@
+cargo run --package sorter2-server -- replay-index
+
diff --git a/server/Cargo.toml b/server/Cargo.toml
index 27f552c20b97ef28cdde4cb6b1a4980375135111..ad4912791aff59fb1d3293f66ad381ae618cd60b 100644
--- a/server/Cargo.toml
+++ b/server/Cargo.toml
@@ -24,6 +24,7 @@ async-stream = "0.3"
futures-util = { version = "0.3", default-features = false, features = ["std"] }
rand = "0.8"
urlencoding = "2"
+url = "2"
durable = { path = "../durable" }
[dev-dependencies]
diff --git a/server/src/entity_store.rs b/server/src/entity_store.rs
index d5d17c3676e4a8ddec998e9f5a9dbafe9c2d9d0e..d29f39aecca6f12cdcf263cf77c3654eb4ee6cfa 100644
--- a/server/src/entity_store.rs
+++ b/server/src/entity_store.rs
@@ -124,7 +124,7 @@ mod tests {
fn round_trip_payload() {
let tmp = tempfile::tempdir().unwrap();
let store = EntityStore::open(tmp.path()).unwrap();
- let id = ItemId::parse("reddit.com/r/rust").unwrap();
+ let id = ItemId::from_url("https://reddit.com/r/rust").unwrap();
let payload = json!({"kind": "t5", "data": {"display_name": "rust"}});
store.put(&id, &payload).unwrap();
diff --git a/server/src/event_log.rs b/server/src/event_log.rs
index 36f5b406084065b608735987cdb483c236e03081..2c9290b6fdbf2c2ad1c0f1ffd7374b2d9cc97f36 100644
--- a/server/src/event_log.rs
+++ b/server/src/event_log.rs
@@ -199,7 +199,7 @@ mod tests {
log.append(&sample_record(
1,
Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
},
))
.await
@@ -237,7 +237,7 @@ mod tests {
let path = tmp.path().join("events.jsonl");
let log = EventLog::new(&path);
let event = Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
};
log.append(&sample_record(1, event)).await.unwrap();
@@ -255,7 +255,7 @@ mod tests {
let path = tmp.path().join("events.jsonl");
std::fs::write(
&path,
- r#"{"type":"node_ensured","id":"reddit.com/r/rust"}
+ r#"{"type":"node_ensured","id":"https://reddit.com/r/rust"}
{"schema":1,"seq":1,"ts":1,"event":{"type":"vote_recorded","ts":1,"a":"a","b":"b","ratio_left":2,"ratio_right":1,"scope":""}}
"#,
)
@@ -295,7 +295,7 @@ mod tests {
log.append(&sample_record(
1,
Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
},
))
.await
@@ -303,7 +303,7 @@ mod tests {
log.append(&sample_record(
3,
Event::NodeEnsured {
- id: "reddit.com/r/python".into(),
+ id: "https://reddit.com/r/python".into(),
},
))
.await
diff --git a/server/src/journal.rs b/server/src/journal.rs
index 521a108019de1ea870d14c4fafbfe572c20ce0de..50bc89f976edb82b7b0e49e954a8eccbbe82bf87 100644
--- a/server/src/journal.rs
+++ b/server/src/journal.rs
@@ -141,10 +141,10 @@ mod tests {
let j2 = journal.clone();
let (r1, r2) = tokio::join!(
j1.append(Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
}),
j2.append(Event::NodeEnsured {
- id: "reddit.com/r/python".into(),
+ id: "https://reddit.com/r/python".into(),
}),
);
r1.unwrap();
@@ -153,10 +153,10 @@ mod tests {
assert_eq!(projection_store.last_applied_event_count().unwrap(), 2);
let tree = projection_store.load_tree().unwrap();
assert!(tree
- .get(&ItemId::parse("reddit.com/r/rust").unwrap())
+ .get(&ItemId::parse("https://reddit.com/r/rust").unwrap())
.is_some());
assert!(tree
- .get(&ItemId::parse("reddit.com/r/python").unwrap())
+ .get(&ItemId::parse("https://reddit.com/r/python").unwrap())
.is_some());
}
@@ -170,7 +170,7 @@ mod tests {
1,
1,
Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
},
))
.await
@@ -186,7 +186,7 @@ mod tests {
1,
1,
Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
},
)],
)
@@ -202,7 +202,7 @@ mod tests {
);
journal
.append(Event::NodeEnsured {
- id: "reddit.com/r/python".into(),
+ id: "https://reddit.com/r/python".into(),
})
.await
.unwrap();
@@ -227,13 +227,13 @@ mod tests {
journal
.append_many(vec![
Event::NodeEnsured {
- id: "reddit.com/r/rust".into(),
+ id: "https://reddit.com/r/rust".into(),
},
Event::NodeEnsured {
- id: "reddit.com/r/python".into(),
+ id: "https://reddit.com/r/python".into(),
},
Event::NodeEnsured {
- id: "reddit.com/r/clojure".into(),
+ id: "https://reddit.com/r/clojure".into(),
},
])
.await
@@ -245,7 +245,7 @@ mod tests {
assert_eq!(projection_store.last_applied_event_count().unwrap(), 3);
let tree = projection_store.load_tree().unwrap();
assert!(tree
- .get(&ItemId::parse("reddit.com/r/clojure").unwrap())
+ .get(&ItemId::parse("https://reddit.com/r/clojure").unwrap())
.is_some());
}
}
diff --git a/server/src/lib.rs b/server/src/lib.rs
index 9bd5f76fd1406b9b1be4c272f4ba8647edde2678..5c02c8e704e4664453bad75d819df8a067668176 100644
--- a/server/src/lib.rs
+++ b/server/src/lib.rs
@@ -9,6 +9,7 @@ pub mod journal;
pub mod pair;
pub mod parser;
pub mod path_types;
+pub mod url_rules;
pub mod projection_apply;
pub mod projection_store;
pub mod ranking;
diff --git a/server/src/pair.rs b/server/src/pair.rs
index 43f780ba6ea6ce1cdc2e1f4cbb252ba8a10684b9..815a97b80e3e9f348e0937a4f147f2862018edb0 100644
--- a/server/src/pair.rs
+++ b/server/src/pair.rs
@@ -381,42 +381,42 @@ mod tests {
#[test]
fn suggest_prefers_unvoted_pair() {
- let parent = ItemId::parse("reddit.com/r/rust").unwrap();
+ let parent = ItemId::parse("https://reddit.com/r/rust").unwrap();
let mut tree = seed_children(
&parent,
&[
- "reddit.com/r/rust/a",
- "reddit.com/r/rust/b",
- "reddit.com/r/rust/c",
+ "https://reddit.com/r/rust/a",
+ "https://reddit.com/r/rust/b",
+ "https://reddit.com/r/rust/c",
],
);
let vote =
- VoteData::from_recorded(1, "reddit.com/r/rust/a", "reddit.com/r/rust/b", 2, 1).unwrap();
+ VoteData::from_recorded(1, "https://reddit.com/r/rust/a", "https://reddit.com/r/rust/b", 2, 1).unwrap();
tree.apply_vote(&parent, vote);
let group = tree.get(&parent).unwrap().local_ranking.clone();
let pool = children_of(&tree, &parent);
let (l, r) = suggest_next_pair_in_pool(&group, &pool, None).unwrap();
- let voted_ab = (l.as_str() == "reddit.com/r/rust/a" && r.as_str() == "reddit.com/r/rust/b")
- || (l.as_str() == "reddit.com/r/rust/b" && r.as_str() == "reddit.com/r/rust/a");
+ let voted_ab = (l.as_str() == "https://reddit.com/r/rust/a" && r.as_str() == "https://reddit.com/r/rust/b")
+ || (l.as_str() == "https://reddit.com/r/rust/b" && r.as_str() == "https://reddit.com/r/rust/a");
assert!(!voted_ab);
}
#[test]
fn suggest_bridges_separate_components() {
- let parent = ItemId::parse("reddit.com/r/rust").unwrap();
+ let parent = ItemId::parse("https://reddit.com/r/rust").unwrap();
let mut tree = seed_children(
&parent,
&[
- "reddit.com/r/rust/a",
- "reddit.com/r/rust/b",
- "reddit.com/r/rust/c",
- "reddit.com/r/rust/d",
+ "https://reddit.com/r/rust/a",
+ "https://reddit.com/r/rust/b",
+ "https://reddit.com/r/rust/c",
+ "https://reddit.com/r/rust/d",
],
);
let ab =
- VoteData::from_recorded(1, "reddit.com/r/rust/a", "reddit.com/r/rust/b", 2, 1).unwrap();
+ VoteData::from_recorded(1, "https://reddit.com/r/rust/a", "https://reddit.com/r/rust/b", 2, 1).unwrap();
let cd =
- VoteData::from_recorded(2, "reddit.com/r/rust/c", "reddit.com/r/rust/d", 2, 1).unwrap();
+ VoteData::from_recorded(2, "https://reddit.com/r/rust/c", "https://reddit.com/r/rust/d", 2, 1).unwrap();
tree.apply_vote(&parent, ab);
tree.apply_vote(&parent, cd);
let group = tree.get(&parent).unwrap().local_ranking.clone();
@@ -424,37 +424,37 @@ mod tests {
let pair = suggest_next_pair_in_pool(&group, &pool, None).unwrap();
let chosen = pair_set(&pair);
let from_ab =
- chosen.contains("reddit.com/r/rust/a") || chosen.contains("reddit.com/r/rust/b");
+ chosen.contains("https://reddit.com/r/rust/a") || chosen.contains("https://reddit.com/r/rust/b");
let from_cd =
- chosen.contains("reddit.com/r/rust/c") || chosen.contains("reddit.com/r/rust/d");
+ chosen.contains("https://reddit.com/r/rust/c") || chosen.contains("https://reddit.com/r/rust/d");
assert!(from_ab && from_cd, "expected bridge pair, got {:?}", chosen);
}
#[test]
fn suggest_prefers_attach_over_isolate_pair_among_many_unranked() {
- let parent = ItemId::parse("reddit.com/r/rust").unwrap();
+ let parent = ItemId::parse("https://reddit.com/r/rust").unwrap();
let mut tree = seed_children(
&parent,
&[
- "reddit.com/r/rust/a",
- "reddit.com/r/rust/b",
- "reddit.com/r/rust/c",
- "reddit.com/r/rust/d",
- "reddit.com/r/rust/e",
+ "https://reddit.com/r/rust/a",
+ "https://reddit.com/r/rust/b",
+ "https://reddit.com/r/rust/c",
+ "https://reddit.com/r/
… preview truncated; 51,799 characters omittedHardlinks — judgments / attempts / prompt
judgments
attempts
- ~anthropic/claude-sonnet-latest #1
- ~anthropic/claude-sonnet-latest #2
- ~x-ai/grok-latest #1
- openai/gpt-chat-latest #1
Prompt text is loaded only by the download route.