Commit A introduces substantial new functionality and parser behavior across multiple modules. It adds deterministic typed block masking, prose tokenization for item references, URL and external reference handling, stricter DSL parsing rules for braced bodies versus code fences, improved linkification, breadcrumb fixes, resolver updates, and extensive automated tests covering the new behavior. These changes affect core parsing, rendering, and external item support. Commit B is primarily a UI/layout cleanup with some CSS improvements for ranking lists and removal of an extra wrapper element in the compare page. While useful, it is much smaller in scope and impact than the parser and linkification work in Commit A.
constitution · epochs · watch · epoch 3
comparison
c_11ce057e37af (tommy-mor) vs c_e4fb43f04791 (tommy-mor)
download prompt · raw event · cmp_1b21b9cfb46c31
council reasoning
Side A introduces substantial new functionality: deterministic and typed block masking, a full prose tokenizer for item references, improved URL parsing (including punctuation trimming and newline handling), stricter DSL validation rules, and integration into HTML linkification. It also adds extensive tests and refactors multiple components to support these features. Side B is comparatively minor, mainly removing a wrapper element in a template and adding CSS refinements. The scope, complexity, and impact of A are significantly greater.
Side A introduces substantial functional changes across the DSL parser, block masking system, prose tokenization, linkification logic, external resolver behavior, and adds extensive new tests. It improves determinism, enforces stricter parsing rules, enhances URL/item reference handling, and refactors HTML link generation. Side B mainly removes a wrapper shell in one template and adjusts CSS styling for ranking lists, which is comparatively minor and largely presentational. Therefore, Side A contributed significantly more.
sides
A — c_11ce057e37af (tommy-mor)
message
[08565bea] Tokenize prose refs for garden URL links (#147) * Tokenize prose refs for garden URL links Co-authored-by: tommy <thmorriss@gmail.com> * Stop prose URLs at line boundaries Co-authored-by: tommy <thmorriss@gmail.com> * Require braced DSL item bodies Co-authored-by: tommy <thmorriss@gmail.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
diff preview
diff --git a/server/src/dsl.rs b/server/src/dsl.rs
index 4203c2f59dd8825d7a91513c4023f3ccd37f1efc..b3c785674560a0b42a7f4c3aa13d80cd350f485b 100644
--- a/server/src/dsl.rs
+++ b/server/src/dsl.rs
@@ -1,7 +1,5 @@
use std::collections::HashMap;
-use rand::Rng;
-
/// Parsed DSL document.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Document {
@@ -39,31 +37,57 @@ pub enum DslError {
/// Matches the legacy Python parser behavior:
/// - Supports toggle markers (open == close), e.g. ```...```
/// - Supports nested markers (open != close), e.g. { ... { ... } ... }
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum BlockKind {
+ CodeFence,
+ DoubleBrace,
+ Brace,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+struct MaskedBlock {
+ kind: BlockKind,
+}
+
#[derive(Debug, Default, Clone)]
pub struct BlockMasker {
pub replacements: HashMap<String, String>,
+ blocks: HashMap<String, MaskedBlock>,
+ next_id: u32,
}
impl BlockMasker {
pub fn new() -> Self {
Self {
replacements: HashMap::new(),
+ blocks: HashMap::new(),
+ next_id: 0,
}
}
- fn new_token(&mut self) -> String {
- let mut rng = rand::thread_rng();
- let n: u32 = rng.gen();
- let token = format!("__BLOCK_{:08x}__", n);
- // Extremely unlikely collision; if it happens, regenerate.
- if self.replacements.contains_key(&token) {
- return self.new_token();
+ fn new_token(&mut self, haystack: &str) -> String {
+ loop {
+ let token = format!("__BLOCK_{:08x}__", self.next_id);
+ self.next_id = self.next_id.wrapping_add(1);
+ if !self.replacements.contains_key(&token) && !haystack.contains(&token) {
+ return token;
+ }
}
- token
}
/// Replace outermost balanced blocks with tokens.
pub fn mask(&mut self, text: &str, open_marker: &str, close_marker: &str) -> String {
+ self.mask_kind(text, open_marker, close_marker, BlockKind::Brace)
+ }
+
+ /// Replace outermost balanced blocks with typed deterministic tokens.
+ pub fn mask_kind(
+ &mut self,
+ text: &str,
+ open_marker: &str,
+ close_marker: &str,
+ kind: BlockKind,
+ ) -> String {
if text.is_empty() {
return text.to_string();
}
@@ -97,9 +121,10 @@ impl BlockMasker {
// Found end of outermost block
let s = start_idx.max(0) as usize;
let original_block = &text[s..i];
- let token = self.new_token();
+ let token = self.new_token(text);
self.replacements
.insert(token.clone(), original_block.to_string());
+ self.blocks.insert(token.clone(), MaskedBlock { kind });
result_parts.push(token);
current_idx = i;
}
@@ -176,13 +201,22 @@ impl BlockMasker {
}
token.to_string()
}
+
+ pub fn block_kind(&self, token: &str) -> Option<BlockKind> {
+ self.blocks.get(token).map(|b| b.kind)
+ }
}
fn mask_all(mut masker: BlockMasker, text: &str) -> (BlockMasker, String) {
// Mask hierarchy: Code -> Double Brace -> Single Brace.
- let t = masker.mask(text, "```", "```");
- let t = masker.mask(&t, "{{", "}}");
- let t = masker.mask(&t, "{", "}");
+ let t = masker.mask_kind(text, "```", "```", BlockKind::CodeFence);
+ let t = masker.mask_kind(&t, "{{", "}}", BlockKind::DoubleBrace);
+ let t = masker.mask_kind(&t, "{", "}", BlockKind::Brace);
+ (masker, t)
+}
+
+fn mask_code_fences(mut masker: BlockMasker, text: &str) -> (BlockMasker, String) {
+ let t = masker.mask_kind(text, "```", "```", BlockKind::CodeFence);
(masker, t)
}
@@ -253,7 +287,34 @@ fn skip_ws(s: &str, mut i: usize) -> usize {
i
}
-fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ProseToken {
+ Text(String),
+ ItemRef(String),
+}
+
+fn trim_prose_item_ref_end(s: &str, mut end: usize) -> usize {
+ while end > 0 {
+ let Some((idx, c)) = s[..end].char_indices().next_back() else {
+ break;
+ };
+ if matches!(
+ c,
+ '.' | ',' | ';' | ':' | '!' | '?' | ')' | ']' | '}' | '"' | '\''
+ ) {
+ end = idx;
+ } else {
+ break;
+ }
+ }
+ end
+}
+
+fn parse_item_name_at_with_mode(
+ s: &str,
+ i: usize,
+ trim_trailing_punctuation: bool,
+) -> Option<(String, usize)> {
let bytes = s.as_bytes();
if i >= bytes.len() {
return None;
@@ -263,6 +324,9 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
if s[i..].starts_with("https://") || s[i..].starts_with("http://") {
let mut j = i;
while j < bytes.len() {
+ if trim_trailing_punctuation && bytes[j] == b'\n' {
+ break;
+ }
if bytes[j..].starts_with(b"__BLOCK_") || is_ws_byte(bytes[j]) {
break;
}
@@ -271,6 +335,12 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
if j <= i {
return None;
}
+ if trim_trailing_punctuation {
+ j = trim_prose_item_ref_end(s, j);
+ if j <= i {
+ return None;
+ }
+ }
return Some((s[i..j].to_string(), j));
}
@@ -296,6 +366,12 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
if j <= i + 2 {
return None;
}
+ if trim_trailing_punctuation {
+ j = trim_prose_item_ref_end(s, j);
+ if j <= i + 2 {
+ return None;
+ }
+ }
let raw = &s[i..j];
if !is_item_name(raw) {
return None;
@@ -336,6 +412,46 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
Some((format!("~/{}", name), j))
}
+fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
+ parse_item_name_at_with_mode(s, i, false)
+}
+
+pub fn parse_prose_item_ref_at(s: &str, i: usize) -> Option<(String, usize)> {
+ parse_item_name_at_with_mode(s, i, true)
+}
+
+pub fn tokenize_prose_item_refs(text: &str) -> Vec<ProseToken> {
+ if text.is_empty() {
+ return Vec::new();
+ }
+ let (masker, masked) = mask_code_fences(BlockMasker::new(), text);
+ let mut tokens = Vec::new();
+ let mut text_start = 0usize;
+ let mut i = 0usize;
+
+ while i < masked.len() {
+ if let Some((raw, end)) = parse_prose_item_ref_at(&masked, i) {
+ if text_start < i {
+ tokens.push(ProseToken::Text(masker.unmask(&masked[text_start..i])));
+ }
+ tokens.push(ProseToken::ItemRef(masker.unmask(&raw)));
+ i = end;
+ text_start = i;
+ continue;
+ }
+
+ let Some((_, c)) = masked[i..].char_indices().next() else {
+ break;
+ };
+ i += c.len_utf8();
+ }
+
+ if text_start < masked.len() {
+ tokens.push(ProseToken::Text(masker.unmask(&masked[text_start..])));
+ }
+ tokens
+}
+
fn parse_block_token_at(s: &str, i: usize) -> Option<(String, usize)> {
let bytes = s.as_bytes();
if i >= bytes.len() {
@@ -401,6 +517,12 @@ fn parse_block_prefixed_statement(
tail: &str,
masker: &BlockMasker,
) -> Result<Stmt, DslError> {
+ if masker.block_kind(block_token) == Some(BlockKind::CodeFence) {
+ return Err(DslError::Parse(
+ "vote explanations must use `{ ... }`; code fences belong inside body blocks"
+ .to_string(),
+ ));
+ }
// vote: block item_ref comparison item_ref
let s = tail.trim_start();
if s.is_empty() {
@@ -462,6 +584,11 @@ fn parse_item_definition_statement(stripped: &str, masker: &BlockMasker) -> Resu
}
if let Some((tok, end)) = parse_block_token_at(stripped, i) {
+ if masker.block_kind(&tok) == Some(BlockKind::CodeFence) {
+ return Err(DslError::Parse(
+ "item bodies must use `{ ... }`; code fences belong inside body blocks".to_string(),
+ ));
+ }
let body = masker.extract_body(&tok);
let tail = stripped[end..].trim();
if !tail.is_empty() {
@@ -572,6 +699,10 @@ pub fn parse_full(text: &str) -> Result<Document, DslError> {
if let Some((tok, end)) = parse_block_token_at(stripped, 0) {
if stripped[end..].trim().is_empty() {
+ if masker.block_kind(&tok) == Some(BlockKind::CodeFence) {
+ prose_buffer.push(line);
+ continue;
+ }
pending_block = Some(tok);
continue;
}
@@ -619,6 +750,70 @@ mod tests {
assert_eq!(roundtrip, input);
}
+ #[test]
+ fn blockmasker_tokens_are_deterministic_and_typed() {
+ let input = "x ```code``` y {body}";
+ let (masker, masked) = mask_all(BlockMasker::new(), input);
+ assert!(masked.contains("__BLOCK_00000000__"));
+ assert!(masked.contains("__BLOCK_00000001__"));
+ assert_eq!(
+ masker.block_kind("__BLOCK_00000000__"),
+ Some(BlockKind::CodeFence)
+ );
+ assert_eq!(
+ masker.block_kind("__BLOCK_00000001__"),
+ Some(BlockKind::Brace)
+ );
+ assert_eq!(masker.unmask(&masked), input);
+ }
+
+ #[test]
+ fn prose_tokenizer_finds_tilde_dash_and_raw_url_refs() {
+ let tokens =
+ tokenize_prose_item_refs("see ~/a/b then -/example.com/x and https://Example.com/A/B.");
+ assert_eq!(
+ tokens,
+ vec![
+ ProseToken::Text("see ".to_string()),
+ ProseToken::ItemRef("~/a/b".to_string()),
+ ProseToken::Text(" then ".to_string()),
+ ProseToken::ItemRef("-/example.com/x".to_string()),
+ ProseToken::Text(" and ".to_string()),
+ ProseToken::ItemRef("https://Example.com/A/B".to_string()),
+ ProseToken::Text(".".to_string()),
+ ]
+ );
+ }
+
+ #[test]
+ fn prose_tokenizer_stops_raw_urls_at_newlines() {
+ let tokens = tokenize_prose_item_refs("https://example.com/a/b.\n-/example.com/a/b");
+ assert_eq!(
+ tokens,
+ vec![
+ ProseToken::ItemRef("https://example.com/a/b".to_string()),
+ ProseToken::Text(".\n".to_string()),
+ ProseToken::ItemRef("-/example.com/a/b".to_string()),
+ ]
+ );
+ }
+
+ #[test]
+ fn prose_tokenizer_does_not_linkify_inside_code_fences() {
+ let tokens = tokenize_prose_item_refs(
+ "before ```json\n{\"url\":\"https://example.com\"}\n``` after ~/x",
+ );
+ assert_eq!(
+ tokens,
+ vec![
+ ProseToken::Text(
+ "before ```json\n{\"url\":\"https://example.com\"}\n``` after ".to_string()
+ ),
+ ProseToken::ItemRef("~/x".to_string()),
+ ]
+ );
+ }
+
#[test]
fn parse_item_with_body_strips_outer_braces() {
let input = "~/rust { Systems language }";
@@ -633,8 +828,8 @@ mod tests {
}
#[test]
- fn parse_item_with_fenced_json_body_preserves_braces() {
- let input = "~/item/in/url ```json\n{\"test\": true}\n```";
+ fn parse_item_with_braced_fenced_json_body_preserves_braces() {
+ let input = "~/item/in/url {\n```json\n{\"test\": true}\n```\n}";
let doc = parse_full(input).unwrap();
assert_eq!(
doc.statements,
@@ -645,6 +840,51 @@ mod tests {
… preview truncated; 18,752 characters omittedB — c_e4fb43f04791 (tommy-mor)
message
[09842c93] remove shell
diff preview
diff --git a/server/src/html/garden.rs b/server/src/html/garden.rs
index 8fc6be1de8dd975f9547de615809222236be4b70..76ba96e45c9e16291a6ccd8096fdd01b274c1560 100644
--- a/server/src/html/garden.rs
+++ b/server/src/html/garden.rs
@@ -1321,55 +1321,53 @@ async fn vote_compare_inner(
.expect("vote compare rpc json");
let body = html! {
- section class="vote-compare-shell" {
- h2 { "compare" }
- div class="vote-compare-pair" {
- a class="vote-compare-item" href=(nav.garden_item_href(&left)) {
- code { (item_display_path(left.as_str())) }
- }
- span class="vote-compare-vs" { "vs" }
- a class="vote-compare-item" href=(nav.garden_item_href(&right)) {
- code { (item_display_path(right.as_str())) }
- }
- }
- div id="vote-edge-history-region" {
- (edge_history)
- }
- @if can_post {
- form id="vote-compare-form" method="POST" action="/ui" {
- input type="hidden" name=(UI_RPC_FIELD) value=(rpc_json);
- div class="vote-thread-picker" {
- label class="vote-thread-picker-label" { "thread" }
- select id="vote-thread-select" name="thread_tag" aria-label="Thread to post vote into" {
- @if thread_tags.is_empty() {
- option value="vote" selected { "#vote" }
- }
- @for t in &thread_tags {
- @if *t == auto_thread {
- option value=(t) selected { "#" (t) }
- } @else {
- option value=(t) { "#" (t) }
- }
- }
- }
+ h2 { "compare" }
+ div class="vote-compare-pair" {
+ a class="vote-compare-item" href=(nav.garden_item_href(&left)) {
+ code { (item_display_path(left.as_str())) }
+ }
+ span class="vote-compare-vs" { "vs" }
+ a class="vote-compare-item" href=(nav.garden_item_href(&right)) {
+ code { (item_display_path(right.as_str())) }
+ }
+ }
+ div id="vote-edge-history-region" {
+ (edge_history)
+ }
+ @if can_post {
+ form id="vote-compare-form" method="POST" action="/ui" {
+ input type="hidden" name=(UI_RPC_FIELD) value=(rpc_json);
+ div class="vote-thread-picker" {
+ label class="vote-thread-picker-label" { "thread" }
+ select id="vote-thread-select" name="thread_tag" aria-label="Thread to post vote into" {
+ @if thread_tags.is_empty() {
+ option value="vote" selected { "#vote" }
}
- input type="hidden" name="ratio_left" id="vote-ratio-left" value="50";
- input type="hidden" name="ratio_right" id="vote-ratio-right" value="50";
- label class="vote-compare-slider-label" {
- span id="vote-slider-left-label" { (item_display_path(left.as_str())) }
- input type="range" id="vote-preference-slider" min="0" max="100" value="50"
- aria-valuemin="0" aria-valuemax="100";
- span id="vote-slider-right-label" { (item_display_path(right.as_str())) }
+ @for t in &thread_tags {
+ @if *t == auto_thread {
+ option value=(t) selected { "#" (t) }
+ } @else {
+ option value=(t) { "#" (t) }
+ }
}
- label class="vote-explain-label" { "reason (required)" }
- textarea name="explanation" id="vote-explain" rows="5" placeholder="why this split?" required {}
- div id="vote-compare-errors" {}
- p { button type="submit" { "post vote" } }
}
- } @else {
- p class="muted" { a href="/login" { "log in" } " to post this vote." }
}
+ input type="hidden" name="ratio_left" id="vote-ratio-left" value="50";
+ input type="hidden" name="ratio_right" id="vote-ratio-right" value="50";
+ label class="vote-compare-slider-label" {
+ span id="vote-slider-left-label" { (item_display_path(left.as_str())) }
+ input type="range" id="vote-preference-slider" min="0" max="100" value="50"
+ aria-valuemin="0" aria-valuemax="100";
+ span id="vote-slider-right-label" { (item_display_path(right.as_str())) }
+ }
+ label class="vote-explain-label" { "reason (required)" }
+ textarea name="explanation" id="vote-explain" rows="5" placeholder="why this split?" required {}
+ div id="vote-compare-errors" {}
+ p { button type="submit" { "post vote" } }
}
+ } @else {
+ p class="muted" { a href="/login" { "log in" } " to post this vote." }
+ }
};
let page = layout_full_bleed_chromeless(
diff --git a/server/static/theme_default.css b/server/static/theme_default.css
index a441c4f79d8cf88f5a8240f9992f47dbbd1ab46b..fdcde86c718eb53cce8273e45a9844c2d8041f88 100644
--- a/server/static/theme_default.css
+++ b/server/static/theme_default.css
@@ -1270,8 +1270,12 @@ body.view-ontology-dark .ont-ranking-list li {
body.view-ontology-dark .ont-ranking-list li::before {
color: var(--meta);
content: counter(ont-rank) ".";
- font-size: 11px;
- min-width: 18px;
+ flex-shrink: 0;
+ font-size: 1.35rem;
+ font-weight: 700;
+ font-variant-numeric: tabular-nums;
+ line-height: 1;
+ min-width: 2.25ch;
text-align: right;
}
body.view-ontology-dark .ont-rank-score {
@@ -1424,8 +1428,12 @@ body.view-ontology-light .ont-ranking-list li {
body.view-ontology-light .ont-ranking-list li::before {
color: var(--meta);
content: counter(ont-rank) ".";
- font-size: 11px;
- min-width: 18px;
+ flex-shrink: 0;
+ font-size: 1.35rem;
+ font-weight: 700;
+ font-variant-numeric: tabular-nums;
+ line-height: 1;
+ min-width: 2.25ch;
text-align: right;
}
body.view-ontology-light .ont-rank-score {
diff --git a/server/static/theme_retro.css b/server/static/theme_retro.css
index 373f15bbd75b62604cdd14b9f0fadda2d6176991..61e1448b2f66a075c0e33325d6980448712fc927 100644
--- a/server/static/theme_retro.css
+++ b/server/static/theme_retro.css
@@ -135,6 +135,30 @@ body.view-ontology nav.breadcrumb a:hover {
body.view-ontology nav.breadcrumb a.bc-current { color: #111; font-weight: 600; }
body.view-ontology nav.breadcrumb .bc-sep { color: #888; padding: 0 2px; }
+body.view-ontology ol.ont-ranking-list {
+ counter-reset: ont-rank;
+ list-style: none;
+ margin: 0.5rem 0;
+ padding: 0;
+}
+body.view-ontology ol.ont-ranking-list li {
+ align-items: baseline;
+ counter-increment: ont-rank;
+ display: flex;
+ gap: 0.35rem;
+}
+body.view-ontology ol.ont-ranking-list li::before {
+ flex-shrink: 0;
+ color: #666;
+ content: counter(ont-rank) ".";
+ font-size: 1.35rem;
+ font-weight: 700;
+ font-variant-numeric: tabular-nums;
+ line-height: 1;
+ min-width: 2.25ch;
+ text-align: right;
+}
+
nav.breadcrumb.ont-sibling-nav {
margin-top: 0;
width: 100%;
diff --git a/server/static/theme_retro_craft.css b/server/static/theme_retro_craft.css
index 6eb9222184a8795d67a5d09d41de08c8ac1b148f..7da102040484c887833158a37c307d078205c701 100644
--- a/server/static/theme_retro_craft.css
+++ b/server/static/theme_retro_craft.css
@@ -742,12 +742,32 @@ body.view-ontology button.ont-garden-pin-ico:focus-visible {
outline-offset: 2px;
}
+body.view-ontology ol.ont-ranking-list {
+ counter-reset: ont-rank;
+ list-style: none;
+ margin: 0;
+ padding: 0;
+}
body.view-ontology ol.ont-ranking-list li,
body.view-ontology ul.ont-group-list li {
display: flex;
align-items: baseline;
gap: 0.35rem;
}
+body.view-ontology ol.ont-ranking-list li {
+ counter-increment: ont-rank;
+}
+body.view-ontology ol.ont-ranking-list li::before {
+ flex-shrink: 0;
+ color: #5c574e;
+ content: counter(ont-rank) ".";
+ font-size: 1.35rem;
+ font-weight: 700;
+ font-variant-numeric: tabular-nums;
+ line-height: 1;
+ min-width: 2.25ch;
+ text-align: right;
+}
body.view-ontology ol.ont-ranking-list li .item-link,
body.view-ontology ul.ont-group-list li .item-link {
flex: 1;
Hardlinks — judgments / attempts / prompt
judgments
attempts
Prompt text is loaded only by the download route.