Side A introduces a substantial architectural improvement by moving vote persistence and ranking recomputation into a dedicated settlement worker, adding cached ranking reads (`ranked_items_cached`) so HTTP handlers can use read locks instead of recomputing under write locks. It also removes an obsolete demo counter and simplifies the UI around the real ranking workflow. Side B adds useful DSL parsing and linkification improvements (typed block masking, prose item reference tokenization, braced body enforcement, and URL linkification), but these are more feature-focused and narrower in long-term impact than the concurrency and performance redesign in Side A.
constitution · epochs · watch · epoch 3
comparison
c_4772ee88dbe3 (tommy-mor) vs c_11ce057e37af (tommy-mor)
download prompt · raw event · cmp_a76fd16a70bae2
council reasoning
A introduces a new async settlement pipeline (mpsc worker, batching, and cached score computation) and shifts ranking reads to a cached, read-only path, improving correctness and performance under concurrency; it also removes unused demo-counter code. B adds richer DSL parsing/tokenization and linkification rules, but these are more feature-level enhancements rather than core architectural improvements.
Side B adds substantive DSL and linkification improvements: deterministic, typed BlockMasker tokens (removing random IDs), a new prose tokenizer that correctly stops raw URLs at punctuation/newlines and skips code fences, enforces braced item bodies, and updates HTML linkification to use these tokens with extensive tests. Side A mainly removes the demo counter and refactors ranking to use cached reads plus a settlement worker, which is valuable but narrower in scope than B’s parser, linking, and correctness enhancements.
B fixes concrete, verifiable bugs in URL/DSL parsing (code-fence contamination of link tokenization, missing trailing-punctuation trimming, ambiguous item-body syntax) and backs each fix with new unit tests, plus swaps a fragile random-token masker for a deterministic one. A's core change (an async settlement worker with score caching) is a reasonable architectural idea but ships with no new tests for the worker/batching/error paths, and much of the diff is just removing demo-counter scaffolding rather than adding tested, lasting functionality.
A introduces a real settlement worker (batched disk append + vote apply + ranking recompute) and a cached read path so HTTP ranking no longer needs write locks or on-the-fly score iteration, while deleting the whole demo-counter feature surface. B is solid parser/UX work (deterministic typed block masks, prose item-ref tokenization, braced bodies, garden linkify for URLs/dash refs), but it extends presentation/DSL edges rather than core vote/ranking durability.
sides
A — c_4772ee88dbe3 (tommy-mor)
message
[07715165] nice
diff preview
diff --git a/server/src/api/ui_html.rs b/server/src/api/ui_html.rs
index c3c62a76f424010d77a6090c84dd0b82098f573e..da2536112faea313352624cf2ce0ddd0ab3377c1 100644
--- a/server/src/api/ui_html.rs
+++ b/server/src/api/ui_html.rs
@@ -6,7 +6,7 @@ use axum::{
use std::collections::HashMap;
use crate::{
- html::{demo_counter_panel, js_string_literal, ranking_panel, JsBuilder},
+ html::{js_string_literal, ranking_panel, JsBuilder},
parser::parse_reddit_url,
parser_render::parser_panel_morph,
state::AppState,
@@ -35,13 +35,6 @@ pub async fn post_ui_html(
};
match action {
- HtmlUiAction::BumpDemoCounter => {
- let count = state.bump_demo_counter().await;
- let panel = demo_counter_panel(count, state.event_log.path().to_string_lossy().as_ref());
- JsBuilder::new()
- .morph_selector("#demo-counter-panel", panel)
- .into_response()
- }
HtmlUiAction::RecordVote {
a,
b,
@@ -54,8 +47,8 @@ pub async fn post_ui_html(
{
return ui_js_warn(&e).into_response();
}
- let mut group = state.group.write().await;
- let panel = ranking_panel(&mut group);
+ let group = state.group.read().await;
+ let panel = ranking_panel(&group);
JsBuilder::new()
.morph_selector("#ranking-panel", panel)
.into_response()
@@ -87,20 +80,6 @@ mod tests {
assert!(matches!(err, HtmlUiParseError::MissingRpc));
}
- #[test]
- fn bump_action_deserializes() {
- let template = serde_json::json!({ "action": "bump_demo_counter" });
- let mut form = HashMap::new();
- form.insert(
- UI_RPC_FIELD.to_string(),
- serde_json::to_string(&template).unwrap(),
- );
- assert_eq!(
- parse_html_ui_from_form(&form).unwrap(),
- HtmlUiAction::BumpDemoCounter
- );
- }
-
#[test]
fn record_vote_action_deserializes() {
let template = serde_json::json!({
diff --git a/server/src/events.rs b/server/src/events.rs
index b969242534e184d4f0a689543a479670b08a18df..eff80aef0257f706d2341f666e63d6a3d921bf6e 100644
--- a/server/src/events.rs
+++ b/server/src/events.rs
@@ -5,8 +5,6 @@ use serde::{Deserialize, Serialize};
pub enum Event {
/// Page view recorded (path → counter in views.json).
ViewRecorded { path: String, ts: i64 },
- /// Demo counter bump from `POST /ui` (persisted in the single JSONL log).
- DemoCounterBumped { ts: i64, value: u64 },
/// Pairwise comparison vote (replayed into [`crate::reducer::GroupState`] on boot).
VoteRecorded {
ts: i64,
diff --git a/server/src/html/mod.rs b/server/src/html/mod.rs
index 5b1d0b5a887d89e7e80796aa7a6c8ed5baaf2782..d69ed962b5c8625bc83c933b1825f2cc1d0868e2 100644
--- a/server/src/html/mod.rs
+++ b/server/src/html/mod.rs
@@ -13,7 +13,7 @@ use crate::{
form_template::template_json_compact,
parser_action::ParserAction,
parser_render::parser_panel,
- ranking::ranked_items,
+ ranking::ranked_items_cached,
reducer::GroupState,
state::AppState,
ui_action::UI_RPC_FIELD,
@@ -199,10 +199,8 @@ fn layout(title: &str, body: Markup, views: u64, theme: &str, theme_next: &str)
}
}
-pub fn ranking_panel(group: &mut GroupState) -> Markup {
- const MAX_ITERS: usize = 10_000;
- const TOL: f64 = 1e-8;
- let items = ranked_items(group, MAX_ITERS, TOL);
+pub fn ranking_panel(group: &GroupState) -> Markup {
+ let items = ranked_items_cached(group);
html! {
section id="ranking-panel" class="demo-panel" {
h2 { "Ranking" }
@@ -260,35 +258,6 @@ pub fn vote_panel() -> Markup {
}
-pub fn demo_counter_panel(count: u64, event_log_path: &str) -> Markup {
- let rpc = template_json_compact(&serde_json::json!({ "action": "bump_demo_counter" }))
- .expect("rpc json");
- html! {
- section id="demo-counter-panel" class="demo-panel" {
- h1 { "sorter2" }
- p class="muted" {
- "Pairwise ranking scaffold — votes persist to JSONL and replay on boot."
- }
- p class="demo-count" {
- strong { "Counter: " }
- span id="demo-count-value" { (count) }
- }
- p class="muted small" {
- "Event log: " code { (event_log_path) }
- }
- form method="post" action="/ui" id="demo-bump-form" {
- input type="hidden" name=(UI_RPC_FIELD) value=(rpc);
- button type="submit" class="btn-primary" { "Bump (POST /ui → eval JS)" }
- }
- p class="muted small" {
- "Uses hidden "
- code { "__rpc__" }
- " JSON + Idiomorph morph — no full page reload."
- }
- }
- }
-}
-
pub async fn home(
State(state): State<AppState>,
jar: CookieJar,
@@ -297,16 +266,15 @@ pub async fn home(
let path = uri.path().to_string();
state.views.increment(path.clone());
let views = state.views.get_views(&path);
- let count = *state.demo_counter.read().await;
let theme = theme_from_jar(&jar);
let theme_next = theme_next_from_uri(&uri);
- let mut group = state.group.write().await;
+ let group = state.group.read().await;
let empty_action = ParserAction::suggest(String::new(), None);
let body = html! {
+ h1 { "sorter2" }
(parser_panel("", &empty_action))
(vote_panel())
- (ranking_panel(&mut group))
- (demo_counter_panel(count, state.event_log.path().to_string_lossy().as_ref()))
+ (ranking_panel(&group))
};
layout("sorter2", body, views, theme, &theme_next)
}
diff --git a/server/src/lib.rs b/server/src/lib.rs
index 6716c5b282e7980a7a0f03d63ad8b25eda61cc55..fa423640d598f4ba97a5885d228e78d7b97f7a22 100644
--- a/server/src/lib.rs
+++ b/server/src/lib.rs
@@ -9,6 +9,7 @@ pub mod parser_render;
pub mod path_types;
pub mod ranking;
pub mod reducer;
+pub mod settlement;
pub mod state;
pub mod ui_action;
pub mod views;
diff --git a/server/src/ranking.rs b/server/src/ranking.rs
index 89d3280126a8d8f841721ce8cb63ff735d68752a..2d706762792ba9239bb3f1c2e4974a2fde908013 100644
--- a/server/src/ranking.rs
+++ b/server/src/ranking.rs
@@ -91,6 +91,11 @@ pub fn compute_group_ranking(group: &mut GroupState, max_iters: usize, tol: f64)
pub fn ranked_items(group: &mut GroupState, max_iters: usize, tol: f64) -> Vec<RankedItem> {
compute_group_ranking(group, max_iters, tol);
+ ranked_items_cached(group)
+}
+
+/// Read cached scores without recomputing (HTTP fast path).
+pub fn ranked_items_cached(group: &GroupState) -> Vec<RankedItem> {
let mut items: Vec<RankedItem> = group
.idx_to_item
.iter()
@@ -105,7 +110,12 @@ pub fn ranked_items(group: &mut GroupState, max_iters: usize, tol: f64) -> Vec<R
items
}
-fn compute_scores_from_edges(n: usize, edges: impl Iterator<Item = ((usize, usize), f64)>, max_iters: usize, tol: f64) -> Vec<f64> {
+pub fn compute_scores_from_edges(
+ n: usize,
+ edges: impl Iterator<Item = ((usize, usize), f64)>,
+ max_iters: usize,
+ tol: f64,
+) -> Vec<f64> {
if n == 0 {
return vec![];
}
diff --git a/server/src/settlement.rs b/server/src/settlement.rs
new file mode 100644
index 0000000000000000000000000000000000000000..1f722ceaea62cda22c28ab71551f259fbf049b81
--- /dev/null
+++ b/server/src/settlement.rs
@@ -0,0 +1,114 @@
+use std::sync::Arc;
+
+use tokio::sync::{mpsc, oneshot, RwLock};
+
+use crate::{
+ event_log::EventLog,
+ events::Event,
+ ranking::compute_scores_from_edges,
+ reducer::{GroupState, VoteData},
+};
+
+const MAX_ITERS: usize = 10_000;
+const TOL: f64 = 1e-8;
+
+pub struct SettlementCommand {
+ pub vote: VoteData,
+ pub event: Event,
+ pub reply: oneshot::Sender<Result<(), String>>,
+}
+
+#[derive(Clone)]
+pub struct SettlementClient {
+ tx: mpsc::Sender<SettlementCommand>,
+}
+
+impl SettlementClient {
+ pub fn spawn(group: Arc<RwLock<GroupState>>, event_log: Arc<EventLog>) -> Self {
+ let (tx, rx) = mpsc::channel(64);
+ tokio::spawn(settlement_worker(rx, group, event_log));
+ Self { tx }
+ }
+
+ pub async fn record_vote(&self, vote: VoteData, event: Event) -> Result<(), String> {
+ let (reply, rx) = oneshot::channel();
+ self.tx
+ .send(SettlementCommand {
+ vote,
+ event,
+ reply,
+ })
+ .await
+ .map_err(|_| "settlement worker stopped".to_string())?;
+ rx.await
+ .map_err(|_| "settlement worker stopped".to_string())?
+ }
+}
+
+async fn settlement_worker(
+ mut rx: mpsc::Receiver<SettlementCommand>,
+ group: Arc<RwLock<GroupState>>,
+ event_log: Arc<EventLog>,
+) {
+ while let Some(first) = rx.recv().await {
+ let mut batch = vec![first];
+ while let Ok(more) = rx.try_recv() {
+ batch.push(more);
+ }
+
+ let mut disk_err: Option<String> = None;
+ for cmd in &batch {
+ if let Err(e) = event_log.append(&cmd.event).await {
+ disk_err = Some(e.to_string());
+ break;
+ }
+ }
+
+ if let Some(err) = disk_err {
+ for cmd in batch {
+ let _ = cmd.reply.send(Err(err.clone()));
+ }
+ continue;
+ }
+
+ let (edges, n) = {
+ let mut w = group.write().await;
+ for cmd in &batch {
+ w.apply_vote(cmd.vote.clone());
+ }
+ (w.edges.clone(), w.idx_to_item.len())
+ };
+
+ let new_scores = compute_scores_from_edges(
+ n,
+ edges.iter().map(|(&k, &v)| (k, v)),
+ MAX_ITERS,
+ TOL,
+ );
+
+ {
+ let mut w = group.write().await;
+ w.cached_scores = new_scores;
+ w.dirty = false;
+ }
+
+ for cmd in batch {
+ let _ = cmd.reply.send(Ok(()));
+ }
+ }
+}
+
+/// Compute ranking cache from current in-memory edges (startup replay only).
+pub fn warm_ranking_cache(group: &mut GroupState) {
+ if !group.dirty {
+ return;
+ }
+ let n = group.idx_to_item.len();
+ group.cached_scores = compute_scores_from_edges(
+ n,
+ group.edges.iter().map(|(&k, &v)| (k, v)),
+ MAX_ITERS,
+ TOL,
+ );
+ group.dirty = false;
+}
diff --git a/server/src/state.rs b/server/src/state.rs
index 8ec9902e2ecc31cf8208f7ad6365891dc5537eed..1922541a4064c2de1df2d993a461cae783320e05 100644
--- a/server/src/state.rs
+++ b/server/src/state.rs
@@ -6,6 +6,7 @@ use crate::{
event_log::EventLog,
events::Event,
reducer::{GroupState, VoteData},
+ settlement::{warm_ranking_cache, SettlementClient},
views::ViewStore,
};
@@ -38,8 +39,8 @@ pub struct AppState {
pub cfg: Arc<AppConfig>,
pub event_log: Arc<EventLog>,
pub views: ViewStore,
- pub demo_counter: Arc<RwLock<u64>>,
pub group: Arc<RwLock<GroupState>>,
+ settlement: SettlementClient,
}
impl AppState {
@@ -48,14 +49,10 @@ impl AppState {
let views_path = format!("{}/views.json", cfg.data_dir);
let views = ViewStore::new(&views_path);
- let mut demo_counter: u64 = 0;
let mut group = GroupState::new();
if let Ok((events, _)) = event_log.load_all().await {
for ev in events {
match ev {
- Event::DemoCounterBumped { value, .. } => {
- demo_counter = demo_counter.max(value);
- }
Event::VoteRecorded {
ts,
a,
@@ -74,30 +71,20 @@ impl AppState {
}
}
+
… preview truncated; 6,096 characters omittedB — c_11ce057e37af (tommy-mor)
message
[08565bea] Tokenize prose refs for garden URL links (#147) * Tokenize prose refs for garden URL links Co-authored-by: tommy <thmorriss@gmail.com> * Stop prose URLs at line boundaries Co-authored-by: tommy <thmorriss@gmail.com> * Require braced DSL item bodies Co-authored-by: tommy <thmorriss@gmail.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
diff preview
diff --git a/server/src/dsl.rs b/server/src/dsl.rs
index 4203c2f59dd8825d7a91513c4023f3ccd37f1efc..b3c785674560a0b42a7f4c3aa13d80cd350f485b 100644
--- a/server/src/dsl.rs
+++ b/server/src/dsl.rs
@@ -1,7 +1,5 @@
use std::collections::HashMap;
-use rand::Rng;
-
/// Parsed DSL document.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Document {
@@ -39,31 +37,57 @@ pub enum DslError {
/// Matches the legacy Python parser behavior:
/// - Supports toggle markers (open == close), e.g. ```...```
/// - Supports nested markers (open != close), e.g. { ... { ... } ... }
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum BlockKind {
+ CodeFence,
+ DoubleBrace,
+ Brace,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+struct MaskedBlock {
+ kind: BlockKind,
+}
+
#[derive(Debug, Default, Clone)]
pub struct BlockMasker {
pub replacements: HashMap<String, String>,
+ blocks: HashMap<String, MaskedBlock>,
+ next_id: u32,
}
impl BlockMasker {
pub fn new() -> Self {
Self {
replacements: HashMap::new(),
+ blocks: HashMap::new(),
+ next_id: 0,
}
}
- fn new_token(&mut self) -> String {
- let mut rng = rand::thread_rng();
- let n: u32 = rng.gen();
- let token = format!("__BLOCK_{:08x}__", n);
- // Extremely unlikely collision; if it happens, regenerate.
- if self.replacements.contains_key(&token) {
- return self.new_token();
+ fn new_token(&mut self, haystack: &str) -> String {
+ loop {
+ let token = format!("__BLOCK_{:08x}__", self.next_id);
+ self.next_id = self.next_id.wrapping_add(1);
+ if !self.replacements.contains_key(&token) && !haystack.contains(&token) {
+ return token;
+ }
}
- token
}
/// Replace outermost balanced blocks with tokens.
pub fn mask(&mut self, text: &str, open_marker: &str, close_marker: &str) -> String {
+ self.mask_kind(text, open_marker, close_marker, BlockKind::Brace)
+ }
+
+ /// Replace outermost balanced blocks with typed deterministic tokens.
+ pub fn mask_kind(
+ &mut self,
+ text: &str,
+ open_marker: &str,
+ close_marker: &str,
+ kind: BlockKind,
+ ) -> String {
if text.is_empty() {
return text.to_string();
}
@@ -97,9 +121,10 @@ impl BlockMasker {
// Found end of outermost block
let s = start_idx.max(0) as usize;
let original_block = &text[s..i];
- let token = self.new_token();
+ let token = self.new_token(text);
self.replacements
.insert(token.clone(), original_block.to_string());
+ self.blocks.insert(token.clone(), MaskedBlock { kind });
result_parts.push(token);
current_idx = i;
}
@@ -176,13 +201,22 @@ impl BlockMasker {
}
token.to_string()
}
+
+ pub fn block_kind(&self, token: &str) -> Option<BlockKind> {
+ self.blocks.get(token).map(|b| b.kind)
+ }
}
fn mask_all(mut masker: BlockMasker, text: &str) -> (BlockMasker, String) {
// Mask hierarchy: Code -> Double Brace -> Single Brace.
- let t = masker.mask(text, "```", "```");
- let t = masker.mask(&t, "{{", "}}");
- let t = masker.mask(&t, "{", "}");
+ let t = masker.mask_kind(text, "```", "```", BlockKind::CodeFence);
+ let t = masker.mask_kind(&t, "{{", "}}", BlockKind::DoubleBrace);
+ let t = masker.mask_kind(&t, "{", "}", BlockKind::Brace);
+ (masker, t)
+}
+
+fn mask_code_fences(mut masker: BlockMasker, text: &str) -> (BlockMasker, String) {
+ let t = masker.mask_kind(text, "```", "```", BlockKind::CodeFence);
(masker, t)
}
@@ -253,7 +287,34 @@ fn skip_ws(s: &str, mut i: usize) -> usize {
i
}
-fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ProseToken {
+ Text(String),
+ ItemRef(String),
+}
+
+fn trim_prose_item_ref_end(s: &str, mut end: usize) -> usize {
+ while end > 0 {
+ let Some((idx, c)) = s[..end].char_indices().next_back() else {
+ break;
+ };
+ if matches!(
+ c,
+ '.' | ',' | ';' | ':' | '!' | '?' | ')' | ']' | '}' | '"' | '\''
+ ) {
+ end = idx;
+ } else {
+ break;
+ }
+ }
+ end
+}
+
+fn parse_item_name_at_with_mode(
+ s: &str,
+ i: usize,
+ trim_trailing_punctuation: bool,
+) -> Option<(String, usize)> {
let bytes = s.as_bytes();
if i >= bytes.len() {
return None;
@@ -263,6 +324,9 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
if s[i..].starts_with("https://") || s[i..].starts_with("http://") {
let mut j = i;
while j < bytes.len() {
+ if trim_trailing_punctuation && bytes[j] == b'\n' {
+ break;
+ }
if bytes[j..].starts_with(b"__BLOCK_") || is_ws_byte(bytes[j]) {
break;
}
@@ -271,6 +335,12 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
if j <= i {
return None;
}
+ if trim_trailing_punctuation {
+ j = trim_prose_item_ref_end(s, j);
+ if j <= i {
+ return None;
+ }
+ }
return Some((s[i..j].to_string(), j));
}
@@ -296,6 +366,12 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
if j <= i + 2 {
return None;
}
+ if trim_trailing_punctuation {
+ j = trim_prose_item_ref_end(s, j);
+ if j <= i + 2 {
+ return None;
+ }
+ }
let raw = &s[i..j];
if !is_item_name(raw) {
return None;
@@ -336,6 +412,46 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
Some((format!("~/{}", name), j))
}
+fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
+ parse_item_name_at_with_mode(s, i, false)
+}
+
+pub fn parse_prose_item_ref_at(s: &str, i: usize) -> Option<(String, usize)> {
+ parse_item_name_at_with_mode(s, i, true)
+}
+
+pub fn tokenize_prose_item_refs(text: &str) -> Vec<ProseToken> {
+ if text.is_empty() {
+ return Vec::new();
+ }
+ let (masker, masked) = mask_code_fences(BlockMasker::new(), text);
+ let mut tokens = Vec::new();
+ let mut text_start = 0usize;
+ let mut i = 0usize;
+
+ while i < masked.len() {
+ if let Some((raw, end)) = parse_prose_item_ref_at(&masked, i) {
+ if text_start < i {
+ tokens.push(ProseToken::Text(masker.unmask(&masked[text_start..i])));
+ }
+ tokens.push(ProseToken::ItemRef(masker.unmask(&raw)));
+ i = end;
+ text_start = i;
+ continue;
+ }
+
+ let Some((_, c)) = masked[i..].char_indices().next() else {
+ break;
+ };
+ i += c.len_utf8();
+ }
+
+ if text_start < masked.len() {
+ tokens.push(ProseToken::Text(masker.unmask(&masked[text_start..])));
+ }
+ tokens
+}
+
fn parse_block_token_at(s: &str, i: usize) -> Option<(String, usize)> {
let bytes = s.as_bytes();
if i >= bytes.len() {
@@ -401,6 +517,12 @@ fn parse_block_prefixed_statement(
tail: &str,
masker: &BlockMasker,
) -> Result<Stmt, DslError> {
+ if masker.block_kind(block_token) == Some(BlockKind::CodeFence) {
+ return Err(DslError::Parse(
+ "vote explanations must use `{ ... }`; code fences belong inside body blocks"
+ .to_string(),
+ ));
+ }
// vote: block item_ref comparison item_ref
let s = tail.trim_start();
if s.is_empty() {
@@ -462,6 +584,11 @@ fn parse_item_definition_statement(stripped: &str, masker: &BlockMasker) -> Resu
}
if let Some((tok, end)) = parse_block_token_at(stripped, i) {
+ if masker.block_kind(&tok) == Some(BlockKind::CodeFence) {
+ return Err(DslError::Parse(
+ "item bodies must use `{ ... }`; code fences belong inside body blocks".to_string(),
+ ));
+ }
let body = masker.extract_body(&tok);
let tail = stripped[end..].trim();
if !tail.is_empty() {
@@ -572,6 +699,10 @@ pub fn parse_full(text: &str) -> Result<Document, DslError> {
if let Some((tok, end)) = parse_block_token_at(stripped, 0) {
if stripped[end..].trim().is_empty() {
+ if masker.block_kind(&tok) == Some(BlockKind::CodeFence) {
+ prose_buffer.push(line);
+ continue;
+ }
pending_block = Some(tok);
continue;
}
@@ -619,6 +750,70 @@ mod tests {
assert_eq!(roundtrip, input);
}
+ #[test]
+ fn blockmasker_tokens_are_deterministic_and_typed() {
+ let input = "x ```code``` y {body}";
+ let (masker, masked) = mask_all(BlockMasker::new(), input);
+ assert!(masked.contains("__BLOCK_00000000__"));
+ assert!(masked.contains("__BLOCK_00000001__"));
+ assert_eq!(
+ masker.block_kind("__BLOCK_00000000__"),
+ Some(BlockKind::CodeFence)
+ );
+ assert_eq!(
+ masker.block_kind("__BLOCK_00000001__"),
+ Some(BlockKind::Brace)
+ );
+ assert_eq!(masker.unmask(&masked), input);
+ }
+
+ #[test]
+ fn prose_tokenizer_finds_tilde_dash_and_raw_url_refs() {
+ let tokens =
+ tokenize_prose_item_refs("see ~/a/b then -/example.com/x and https://Example.com/A/B.");
+ assert_eq!(
+ tokens,
+ vec![
+ ProseToken::Text("see ".to_string()),
+ ProseToken::ItemRef("~/a/b".to_string()),
+ ProseToken::Text(" then ".to_string()),
+ ProseToken::ItemRef("-/example.com/x".to_string()),
+ ProseToken::Text(" and ".to_string()),
+ ProseToken::ItemRef("https://Example.com/A/B".to_string()),
+ ProseToken::Text(".".to_string()),
+ ]
+ );
+ }
+
+ #[test]
+ fn prose_tokenizer_stops_raw_urls_at_newlines() {
+ let tokens = tokenize_prose_item_refs("https://example.com/a/b.\n-/example.com/a/b");
+ assert_eq!(
+ tokens,
+ vec![
+ ProseToken::ItemRef("https://example.com/a/b".to_string()),
+ ProseToken::Text(".\n".to_string()),
+ ProseToken::ItemRef("-/example.com/a/b".to_string()),
+ ]
+ );
+ }
+
+ #[test]
+ fn prose_tokenizer_does_not_linkify_inside_code_fences() {
+ let tokens = tokenize_prose_item_refs(
+ "before ```json\n{\"url\":\"https://example.com\"}\n``` after ~/x",
+ );
+ assert_eq!(
+ tokens,
+ vec![
+ ProseToken::Text(
+ "before ```json\n{\"url\":\"https://example.com\"}\n``` after ".to_string()
+ ),
+ ProseToken::ItemRef("~/x".to_string()),
+ ]
+ );
+ }
+
#[test]
fn parse_item_with_body_strips_outer_braces() {
let input = "~/rust { Systems language }";
@@ -633,8 +828,8 @@ mod tests {
}
#[test]
- fn parse_item_with_fenced_json_body_preserves_braces() {
- let input = "~/item/in/url ```json\n{\"test\": true}\n```";
+ fn parse_item_with_braced_fenced_json_body_preserves_braces() {
+ let input = "~/item/in/url {\n```json\n{\"test\": true}\n```\n}";
let doc = parse_full(input).unwrap();
assert_eq!(
doc.statements,
@@ -645,6 +840,51 @@ mod tests {
… preview truncated; 18,752 characters omittedHardlinks — judgments / attempts / prompt
judgments
- openai/gpt-chat-latest: A (3:2)
- openai/gpt-5.3-chat: A (3:2)
- openai/gpt-5.2-chat: B (3:1)
- ~anthropic/claude-sonnet-latest: B (6:4)
- ~x-ai/grok-latest: A (2:1)
attempts
- openai/gpt-chat-latest #1
- openai/gpt-5.3-chat #1
- openai/gpt-5.2-chat #1
- ~anthropic/claude-sonnet-latest #1
- ~x-ai/grok-latest #1
Prompt text is loaded only by the download route.