Commit A makes a substantial architectural change to the server. It removes the temporary demo counter feature, introduces a dedicated asynchronous settlement worker that batches vote processing, separates ranking computation from cached ranking reads, warms the ranking cache at startup, changes locking from write to read on rendering paths, and updates state management and tests accordingly. These changes improve performance, concurrency, and the overall design of the application rather than just removing dead code. Commit B is also significant, adding deterministic typed block tokens, prose item-reference tokenization, support for linkifying raw URL references while avoiding code fences, stricter DSL parsing for braced item bodies, breadcrumb fixes, and extensive tests. It meaningfully improves parsing correctness and user-facing linkification. Both commits are valuable, but A has broader impact on core runtime behavior and application architecture, whereas B is primarily focused on DSL parsing and rendering correctness.
constitution · epochs · watch · epoch 3
comparison
c_4772ee88dbe3 (tommy-mor) vs c_11ce057e37af (tommy-mor)
download prompt · raw event · cmp_6e4136c1089215
council reasoning
Side A introduces a substantial architectural change: removing the demo counter feature, adding an asynchronous settlement worker with batching, decoupling write/read access to shared state, and introducing cached ranking computation. This affects core request handling, persistence, and performance characteristics across the system. Side B is also significant, with improved DSL parsing, tokenization, and linkification plus strong test coverage, but it is more localized to parsing and rendering logic. Overall, A has broader system-level impact.
Side A introduces a substantial architectural change: removing the demo counter feature and adding a new asynchronous settlement worker with batching, cached ranking computation, startup cache warming, API/state refactors, and corresponding test updates. This meaningfully impacts core state management, performance, and concurrency. Side B improves DSL parsing and linkification with deterministic block masking, prose tokenization, stricter parsing rules, and solid tests—valuable but more localized to parsing/rendering. Overall, Side A delivers broader systemic impact.
sides
A — c_4772ee88dbe3 (tommy-mor)
message
[07715165] nice
diff preview
diff --git a/server/src/api/ui_html.rs b/server/src/api/ui_html.rs
index c3c62a76f424010d77a6090c84dd0b82098f573e..da2536112faea313352624cf2ce0ddd0ab3377c1 100644
--- a/server/src/api/ui_html.rs
+++ b/server/src/api/ui_html.rs
@@ -6,7 +6,7 @@ use axum::{
use std::collections::HashMap;
use crate::{
- html::{demo_counter_panel, js_string_literal, ranking_panel, JsBuilder},
+ html::{js_string_literal, ranking_panel, JsBuilder},
parser::parse_reddit_url,
parser_render::parser_panel_morph,
state::AppState,
@@ -35,13 +35,6 @@ pub async fn post_ui_html(
};
match action {
- HtmlUiAction::BumpDemoCounter => {
- let count = state.bump_demo_counter().await;
- let panel = demo_counter_panel(count, state.event_log.path().to_string_lossy().as_ref());
- JsBuilder::new()
- .morph_selector("#demo-counter-panel", panel)
- .into_response()
- }
HtmlUiAction::RecordVote {
a,
b,
@@ -54,8 +47,8 @@ pub async fn post_ui_html(
{
return ui_js_warn(&e).into_response();
}
- let mut group = state.group.write().await;
- let panel = ranking_panel(&mut group);
+ let group = state.group.read().await;
+ let panel = ranking_panel(&group);
JsBuilder::new()
.morph_selector("#ranking-panel", panel)
.into_response()
@@ -87,20 +80,6 @@ mod tests {
assert!(matches!(err, HtmlUiParseError::MissingRpc));
}
- #[test]
- fn bump_action_deserializes() {
- let template = serde_json::json!({ "action": "bump_demo_counter" });
- let mut form = HashMap::new();
- form.insert(
- UI_RPC_FIELD.to_string(),
- serde_json::to_string(&template).unwrap(),
- );
- assert_eq!(
- parse_html_ui_from_form(&form).unwrap(),
- HtmlUiAction::BumpDemoCounter
- );
- }
-
#[test]
fn record_vote_action_deserializes() {
let template = serde_json::json!({
diff --git a/server/src/events.rs b/server/src/events.rs
index b969242534e184d4f0a689543a479670b08a18df..eff80aef0257f706d2341f666e63d6a3d921bf6e 100644
--- a/server/src/events.rs
+++ b/server/src/events.rs
@@ -5,8 +5,6 @@ use serde::{Deserialize, Serialize};
pub enum Event {
/// Page view recorded (path → counter in views.json).
ViewRecorded { path: String, ts: i64 },
- /// Demo counter bump from `POST /ui` (persisted in the single JSONL log).
- DemoCounterBumped { ts: i64, value: u64 },
/// Pairwise comparison vote (replayed into [`crate::reducer::GroupState`] on boot).
VoteRecorded {
ts: i64,
diff --git a/server/src/html/mod.rs b/server/src/html/mod.rs
index 5b1d0b5a887d89e7e80796aa7a6c8ed5baaf2782..d69ed962b5c8625bc83c933b1825f2cc1d0868e2 100644
--- a/server/src/html/mod.rs
+++ b/server/src/html/mod.rs
@@ -13,7 +13,7 @@ use crate::{
form_template::template_json_compact,
parser_action::ParserAction,
parser_render::parser_panel,
- ranking::ranked_items,
+ ranking::ranked_items_cached,
reducer::GroupState,
state::AppState,
ui_action::UI_RPC_FIELD,
@@ -199,10 +199,8 @@ fn layout(title: &str, body: Markup, views: u64, theme: &str, theme_next: &str)
}
}
-pub fn ranking_panel(group: &mut GroupState) -> Markup {
- const MAX_ITERS: usize = 10_000;
- const TOL: f64 = 1e-8;
- let items = ranked_items(group, MAX_ITERS, TOL);
+pub fn ranking_panel(group: &GroupState) -> Markup {
+ let items = ranked_items_cached(group);
html! {
section id="ranking-panel" class="demo-panel" {
h2 { "Ranking" }
@@ -260,35 +258,6 @@ pub fn vote_panel() -> Markup {
}
-pub fn demo_counter_panel(count: u64, event_log_path: &str) -> Markup {
- let rpc = template_json_compact(&serde_json::json!({ "action": "bump_demo_counter" }))
- .expect("rpc json");
- html! {
- section id="demo-counter-panel" class="demo-panel" {
- h1 { "sorter2" }
- p class="muted" {
- "Pairwise ranking scaffold — votes persist to JSONL and replay on boot."
- }
- p class="demo-count" {
- strong { "Counter: " }
- span id="demo-count-value" { (count) }
- }
- p class="muted small" {
- "Event log: " code { (event_log_path) }
- }
- form method="post" action="/ui" id="demo-bump-form" {
- input type="hidden" name=(UI_RPC_FIELD) value=(rpc);
- button type="submit" class="btn-primary" { "Bump (POST /ui → eval JS)" }
- }
- p class="muted small" {
- "Uses hidden "
- code { "__rpc__" }
- " JSON + Idiomorph morph — no full page reload."
- }
- }
- }
-}
-
pub async fn home(
State(state): State<AppState>,
jar: CookieJar,
@@ -297,16 +266,15 @@ pub async fn home(
let path = uri.path().to_string();
state.views.increment(path.clone());
let views = state.views.get_views(&path);
- let count = *state.demo_counter.read().await;
let theme = theme_from_jar(&jar);
let theme_next = theme_next_from_uri(&uri);
- let mut group = state.group.write().await;
+ let group = state.group.read().await;
let empty_action = ParserAction::suggest(String::new(), None);
let body = html! {
+ h1 { "sorter2" }
(parser_panel("", &empty_action))
(vote_panel())
- (ranking_panel(&mut group))
- (demo_counter_panel(count, state.event_log.path().to_string_lossy().as_ref()))
+ (ranking_panel(&group))
};
layout("sorter2", body, views, theme, &theme_next)
}
diff --git a/server/src/lib.rs b/server/src/lib.rs
index 6716c5b282e7980a7a0f03d63ad8b25eda61cc55..fa423640d598f4ba97a5885d228e78d7b97f7a22 100644
--- a/server/src/lib.rs
+++ b/server/src/lib.rs
@@ -9,6 +9,7 @@ pub mod parser_render;
pub mod path_types;
pub mod ranking;
pub mod reducer;
+pub mod settlement;
pub mod state;
pub mod ui_action;
pub mod views;
diff --git a/server/src/ranking.rs b/server/src/ranking.rs
index 89d3280126a8d8f841721ce8cb63ff735d68752a..2d706762792ba9239bb3f1c2e4974a2fde908013 100644
--- a/server/src/ranking.rs
+++ b/server/src/ranking.rs
@@ -91,6 +91,11 @@ pub fn compute_group_ranking(group: &mut GroupState, max_iters: usize, tol: f64)
pub fn ranked_items(group: &mut GroupState, max_iters: usize, tol: f64) -> Vec<RankedItem> {
compute_group_ranking(group, max_iters, tol);
+ ranked_items_cached(group)
+}
+
+/// Read cached scores without recomputing (HTTP fast path).
+pub fn ranked_items_cached(group: &GroupState) -> Vec<RankedItem> {
let mut items: Vec<RankedItem> = group
.idx_to_item
.iter()
@@ -105,7 +110,12 @@ pub fn ranked_items(group: &mut GroupState, max_iters: usize, tol: f64) -> Vec<R
items
}
-fn compute_scores_from_edges(n: usize, edges: impl Iterator<Item = ((usize, usize), f64)>, max_iters: usize, tol: f64) -> Vec<f64> {
+pub fn compute_scores_from_edges(
+ n: usize,
+ edges: impl Iterator<Item = ((usize, usize), f64)>,
+ max_iters: usize,
+ tol: f64,
+) -> Vec<f64> {
if n == 0 {
return vec![];
}
diff --git a/server/src/settlement.rs b/server/src/settlement.rs
new file mode 100644
index 0000000000000000000000000000000000000000..1f722ceaea62cda22c28ab71551f259fbf049b81
--- /dev/null
+++ b/server/src/settlement.rs
@@ -0,0 +1,114 @@
+use std::sync::Arc;
+
+use tokio::sync::{mpsc, oneshot, RwLock};
+
+use crate::{
+ event_log::EventLog,
+ events::Event,
+ ranking::compute_scores_from_edges,
+ reducer::{GroupState, VoteData},
+};
+
+const MAX_ITERS: usize = 10_000;
+const TOL: f64 = 1e-8;
+
+pub struct SettlementCommand {
+ pub vote: VoteData,
+ pub event: Event,
+ pub reply: oneshot::Sender<Result<(), String>>,
+}
+
+#[derive(Clone)]
+pub struct SettlementClient {
+ tx: mpsc::Sender<SettlementCommand>,
+}
+
+impl SettlementClient {
+ pub fn spawn(group: Arc<RwLock<GroupState>>, event_log: Arc<EventLog>) -> Self {
+ let (tx, rx) = mpsc::channel(64);
+ tokio::spawn(settlement_worker(rx, group, event_log));
+ Self { tx }
+ }
+
+ pub async fn record_vote(&self, vote: VoteData, event: Event) -> Result<(), String> {
+ let (reply, rx) = oneshot::channel();
+ self.tx
+ .send(SettlementCommand {
+ vote,
+ event,
+ reply,
+ })
+ .await
+ .map_err(|_| "settlement worker stopped".to_string())?;
+ rx.await
+ .map_err(|_| "settlement worker stopped".to_string())?
+ }
+}
+
+async fn settlement_worker(
+ mut rx: mpsc::Receiver<SettlementCommand>,
+ group: Arc<RwLock<GroupState>>,
+ event_log: Arc<EventLog>,
+) {
+ while let Some(first) = rx.recv().await {
+ let mut batch = vec![first];
+ while let Ok(more) = rx.try_recv() {
+ batch.push(more);
+ }
+
+ let mut disk_err: Option<String> = None;
+ for cmd in &batch {
+ if let Err(e) = event_log.append(&cmd.event).await {
+ disk_err = Some(e.to_string());
+ break;
+ }
+ }
+
+ if let Some(err) = disk_err {
+ for cmd in batch {
+ let _ = cmd.reply.send(Err(err.clone()));
+ }
+ continue;
+ }
+
+ let (edges, n) = {
+ let mut w = group.write().await;
+ for cmd in &batch {
+ w.apply_vote(cmd.vote.clone());
+ }
+ (w.edges.clone(), w.idx_to_item.len())
+ };
+
+ let new_scores = compute_scores_from_edges(
+ n,
+ edges.iter().map(|(&k, &v)| (k, v)),
+ MAX_ITERS,
+ TOL,
+ );
+
+ {
+ let mut w = group.write().await;
+ w.cached_scores = new_scores;
+ w.dirty = false;
+ }
+
+ for cmd in batch {
+ let _ = cmd.reply.send(Ok(()));
+ }
+ }
+}
+
+/// Compute ranking cache from current in-memory edges (startup replay only).
+pub fn warm_ranking_cache(group: &mut GroupState) {
+ if !group.dirty {
+ return;
+ }
+ let n = group.idx_to_item.len();
+ group.cached_scores = compute_scores_from_edges(
+ n,
+ group.edges.iter().map(|(&k, &v)| (k, v)),
+ MAX_ITERS,
+ TOL,
+ );
+ group.dirty = false;
+}
diff --git a/server/src/state.rs b/server/src/state.rs
index 8ec9902e2ecc31cf8208f7ad6365891dc5537eed..1922541a4064c2de1df2d993a461cae783320e05 100644
--- a/server/src/state.rs
+++ b/server/src/state.rs
@@ -6,6 +6,7 @@ use crate::{
event_log::EventLog,
events::Event,
reducer::{GroupState, VoteData},
+ settlement::{warm_ranking_cache, SettlementClient},
views::ViewStore,
};
@@ -38,8 +39,8 @@ pub struct AppState {
pub cfg: Arc<AppConfig>,
pub event_log: Arc<EventLog>,
pub views: ViewStore,
- pub demo_counter: Arc<RwLock<u64>>,
pub group: Arc<RwLock<GroupState>>,
+ settlement: SettlementClient,
}
impl AppState {
@@ -48,14 +49,10 @@ impl AppState {
let views_path = format!("{}/views.json", cfg.data_dir);
let views = ViewStore::new(&views_path);
- let mut demo_counter: u64 = 0;
let mut group = GroupState::new();
if let Ok((events, _)) = event_log.load_all().await {
for ev in events {
match ev {
- Event::DemoCounterBumped { value, .. } => {
- demo_counter = demo_counter.max(value);
- }
Event::VoteRecorded {
ts,
a,
@@ -74,30 +71,20 @@ impl AppState {
}
}
+
… preview truncated; 6,096 characters omittedB — c_11ce057e37af (tommy-mor)
message
[08565bea] Tokenize prose refs for garden URL links (#147) * Tokenize prose refs for garden URL links Co-authored-by: tommy <thmorriss@gmail.com> * Stop prose URLs at line boundaries Co-authored-by: tommy <thmorriss@gmail.com> * Require braced DSL item bodies Co-authored-by: tommy <thmorriss@gmail.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
diff preview
diff --git a/server/src/dsl.rs b/server/src/dsl.rs
index 4203c2f59dd8825d7a91513c4023f3ccd37f1efc..b3c785674560a0b42a7f4c3aa13d80cd350f485b 100644
--- a/server/src/dsl.rs
+++ b/server/src/dsl.rs
@@ -1,7 +1,5 @@
use std::collections::HashMap;
-use rand::Rng;
-
/// Parsed DSL document.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Document {
@@ -39,31 +37,57 @@ pub enum DslError {
/// Matches the legacy Python parser behavior:
/// - Supports toggle markers (open == close), e.g. ```...```
/// - Supports nested markers (open != close), e.g. { ... { ... } ... }
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum BlockKind {
+ CodeFence,
+ DoubleBrace,
+ Brace,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+struct MaskedBlock {
+ kind: BlockKind,
+}
+
#[derive(Debug, Default, Clone)]
pub struct BlockMasker {
pub replacements: HashMap<String, String>,
+ blocks: HashMap<String, MaskedBlock>,
+ next_id: u32,
}
impl BlockMasker {
pub fn new() -> Self {
Self {
replacements: HashMap::new(),
+ blocks: HashMap::new(),
+ next_id: 0,
}
}
- fn new_token(&mut self) -> String {
- let mut rng = rand::thread_rng();
- let n: u32 = rng.gen();
- let token = format!("__BLOCK_{:08x}__", n);
- // Extremely unlikely collision; if it happens, regenerate.
- if self.replacements.contains_key(&token) {
- return self.new_token();
+ fn new_token(&mut self, haystack: &str) -> String {
+ loop {
+ let token = format!("__BLOCK_{:08x}__", self.next_id);
+ self.next_id = self.next_id.wrapping_add(1);
+ if !self.replacements.contains_key(&token) && !haystack.contains(&token) {
+ return token;
+ }
}
- token
}
/// Replace outermost balanced blocks with tokens.
pub fn mask(&mut self, text: &str, open_marker: &str, close_marker: &str) -> String {
+ self.mask_kind(text, open_marker, close_marker, BlockKind::Brace)
+ }
+
+ /// Replace outermost balanced blocks with typed deterministic tokens.
+ pub fn mask_kind(
+ &mut self,
+ text: &str,
+ open_marker: &str,
+ close_marker: &str,
+ kind: BlockKind,
+ ) -> String {
if text.is_empty() {
return text.to_string();
}
@@ -97,9 +121,10 @@ impl BlockMasker {
// Found end of outermost block
let s = start_idx.max(0) as usize;
let original_block = &text[s..i];
- let token = self.new_token();
+ let token = self.new_token(text);
self.replacements
.insert(token.clone(), original_block.to_string());
+ self.blocks.insert(token.clone(), MaskedBlock { kind });
result_parts.push(token);
current_idx = i;
}
@@ -176,13 +201,22 @@ impl BlockMasker {
}
token.to_string()
}
+
+ pub fn block_kind(&self, token: &str) -> Option<BlockKind> {
+ self.blocks.get(token).map(|b| b.kind)
+ }
}
fn mask_all(mut masker: BlockMasker, text: &str) -> (BlockMasker, String) {
// Mask hierarchy: Code -> Double Brace -> Single Brace.
- let t = masker.mask(text, "```", "```");
- let t = masker.mask(&t, "{{", "}}");
- let t = masker.mask(&t, "{", "}");
+ let t = masker.mask_kind(text, "```", "```", BlockKind::CodeFence);
+ let t = masker.mask_kind(&t, "{{", "}}", BlockKind::DoubleBrace);
+ let t = masker.mask_kind(&t, "{", "}", BlockKind::Brace);
+ (masker, t)
+}
+
+fn mask_code_fences(mut masker: BlockMasker, text: &str) -> (BlockMasker, String) {
+ let t = masker.mask_kind(text, "```", "```", BlockKind::CodeFence);
(masker, t)
}
@@ -253,7 +287,34 @@ fn skip_ws(s: &str, mut i: usize) -> usize {
i
}
-fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ProseToken {
+ Text(String),
+ ItemRef(String),
+}
+
+fn trim_prose_item_ref_end(s: &str, mut end: usize) -> usize {
+ while end > 0 {
+ let Some((idx, c)) = s[..end].char_indices().next_back() else {
+ break;
+ };
+ if matches!(
+ c,
+ '.' | ',' | ';' | ':' | '!' | '?' | ')' | ']' | '}' | '"' | '\''
+ ) {
+ end = idx;
+ } else {
+ break;
+ }
+ }
+ end
+}
+
+fn parse_item_name_at_with_mode(
+ s: &str,
+ i: usize,
+ trim_trailing_punctuation: bool,
+) -> Option<(String, usize)> {
let bytes = s.as_bytes();
if i >= bytes.len() {
return None;
@@ -263,6 +324,9 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
if s[i..].starts_with("https://") || s[i..].starts_with("http://") {
let mut j = i;
while j < bytes.len() {
+ if trim_trailing_punctuation && bytes[j] == b'\n' {
+ break;
+ }
if bytes[j..].starts_with(b"__BLOCK_") || is_ws_byte(bytes[j]) {
break;
}
@@ -271,6 +335,12 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
if j <= i {
return None;
}
+ if trim_trailing_punctuation {
+ j = trim_prose_item_ref_end(s, j);
+ if j <= i {
+ return None;
+ }
+ }
return Some((s[i..j].to_string(), j));
}
@@ -296,6 +366,12 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
if j <= i + 2 {
return None;
}
+ if trim_trailing_punctuation {
+ j = trim_prose_item_ref_end(s, j);
+ if j <= i + 2 {
+ return None;
+ }
+ }
let raw = &s[i..j];
if !is_item_name(raw) {
return None;
@@ -336,6 +412,46 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
Some((format!("~/{}", name), j))
}
+fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
+ parse_item_name_at_with_mode(s, i, false)
+}
+
+pub fn parse_prose_item_ref_at(s: &str, i: usize) -> Option<(String, usize)> {
+ parse_item_name_at_with_mode(s, i, true)
+}
+
+pub fn tokenize_prose_item_refs(text: &str) -> Vec<ProseToken> {
+ if text.is_empty() {
+ return Vec::new();
+ }
+ let (masker, masked) = mask_code_fences(BlockMasker::new(), text);
+ let mut tokens = Vec::new();
+ let mut text_start = 0usize;
+ let mut i = 0usize;
+
+ while i < masked.len() {
+ if let Some((raw, end)) = parse_prose_item_ref_at(&masked, i) {
+ if text_start < i {
+ tokens.push(ProseToken::Text(masker.unmask(&masked[text_start..i])));
+ }
+ tokens.push(ProseToken::ItemRef(masker.unmask(&raw)));
+ i = end;
+ text_start = i;
+ continue;
+ }
+
+ let Some((_, c)) = masked[i..].char_indices().next() else {
+ break;
+ };
+ i += c.len_utf8();
+ }
+
+ if text_start < masked.len() {
+ tokens.push(ProseToken::Text(masker.unmask(&masked[text_start..])));
+ }
+ tokens
+}
+
fn parse_block_token_at(s: &str, i: usize) -> Option<(String, usize)> {
let bytes = s.as_bytes();
if i >= bytes.len() {
@@ -401,6 +517,12 @@ fn parse_block_prefixed_statement(
tail: &str,
masker: &BlockMasker,
) -> Result<Stmt, DslError> {
+ if masker.block_kind(block_token) == Some(BlockKind::CodeFence) {
+ return Err(DslError::Parse(
+ "vote explanations must use `{ ... }`; code fences belong inside body blocks"
+ .to_string(),
+ ));
+ }
// vote: block item_ref comparison item_ref
let s = tail.trim_start();
if s.is_empty() {
@@ -462,6 +584,11 @@ fn parse_item_definition_statement(stripped: &str, masker: &BlockMasker) -> Resu
}
if let Some((tok, end)) = parse_block_token_at(stripped, i) {
+ if masker.block_kind(&tok) == Some(BlockKind::CodeFence) {
+ return Err(DslError::Parse(
+ "item bodies must use `{ ... }`; code fences belong inside body blocks".to_string(),
+ ));
+ }
let body = masker.extract_body(&tok);
let tail = stripped[end..].trim();
if !tail.is_empty() {
@@ -572,6 +699,10 @@ pub fn parse_full(text: &str) -> Result<Document, DslError> {
if let Some((tok, end)) = parse_block_token_at(stripped, 0) {
if stripped[end..].trim().is_empty() {
+ if masker.block_kind(&tok) == Some(BlockKind::CodeFence) {
+ prose_buffer.push(line);
+ continue;
+ }
pending_block = Some(tok);
continue;
}
@@ -619,6 +750,70 @@ mod tests {
assert_eq!(roundtrip, input);
}
+ #[test]
+ fn blockmasker_tokens_are_deterministic_and_typed() {
+ let input = "x ```code``` y {body}";
+ let (masker, masked) = mask_all(BlockMasker::new(), input);
+ assert!(masked.contains("__BLOCK_00000000__"));
+ assert!(masked.contains("__BLOCK_00000001__"));
+ assert_eq!(
+ masker.block_kind("__BLOCK_00000000__"),
+ Some(BlockKind::CodeFence)
+ );
+ assert_eq!(
+ masker.block_kind("__BLOCK_00000001__"),
+ Some(BlockKind::Brace)
+ );
+ assert_eq!(masker.unmask(&masked), input);
+ }
+
+ #[test]
+ fn prose_tokenizer_finds_tilde_dash_and_raw_url_refs() {
+ let tokens =
+ tokenize_prose_item_refs("see ~/a/b then -/example.com/x and https://Example.com/A/B.");
+ assert_eq!(
+ tokens,
+ vec![
+ ProseToken::Text("see ".to_string()),
+ ProseToken::ItemRef("~/a/b".to_string()),
+ ProseToken::Text(" then ".to_string()),
+ ProseToken::ItemRef("-/example.com/x".to_string()),
+ ProseToken::Text(" and ".to_string()),
+ ProseToken::ItemRef("https://Example.com/A/B".to_string()),
+ ProseToken::Text(".".to_string()),
+ ]
+ );
+ }
+
+ #[test]
+ fn prose_tokenizer_stops_raw_urls_at_newlines() {
+ let tokens = tokenize_prose_item_refs("https://example.com/a/b.\n-/example.com/a/b");
+ assert_eq!(
+ tokens,
+ vec![
+ ProseToken::ItemRef("https://example.com/a/b".to_string()),
+ ProseToken::Text(".\n".to_string()),
+ ProseToken::ItemRef("-/example.com/a/b".to_string()),
+ ]
+ );
+ }
+
+ #[test]
+ fn prose_tokenizer_does_not_linkify_inside_code_fences() {
+ let tokens = tokenize_prose_item_refs(
+ "before ```json\n{\"url\":\"https://example.com\"}\n``` after ~/x",
+ );
+ assert_eq!(
+ tokens,
+ vec![
+ ProseToken::Text(
+ "before ```json\n{\"url\":\"https://example.com\"}\n``` after ".to_string()
+ ),
+ ProseToken::ItemRef("~/x".to_string()),
+ ]
+ );
+ }
+
#[test]
fn parse_item_with_body_strips_outer_braces() {
let input = "~/rust { Systems language }";
@@ -633,8 +828,8 @@ mod tests {
}
#[test]
- fn parse_item_with_fenced_json_body_preserves_braces() {
- let input = "~/item/in/url ```json\n{\"test\": true}\n```";
+ fn parse_item_with_braced_fenced_json_body_preserves_braces() {
+ let input = "~/item/in/url {\n```json\n{\"test\": true}\n```\n}";
let doc = parse_full(input).unwrap();
assert_eq!(
doc.statements,
@@ -645,6 +840,51 @@ mod tests {
… preview truncated; 18,752 characters omittedHardlinks — judgments / attempts / prompt
judgments
attempts
Prompt text is loaded only by the download route.