A is a focused, well-tested feature/bugfix (deterministic block tokens, proper prose URL tokenization with fence-awareness, breadcrumb root fix) that clearly improves correctness and is validated by new unit tests. B is a sprawling 'first pass' architectural rewrite (REST endpoints collapsed into one giant RPC dispatcher, Thread→Room renaming) that deletes a lot of previously tested, well-structured code and introduces a large, less type-safe generic command enum — valuable functionality-wise but explicitly unfinished/exploratory and far noisier relative to its net design benefit.
constitution · epochs · watch · epoch 3
comparison
c_11ce057e37af (tommy-mor) vs c_2595b6007624 (tommy-mor)
download prompt · raw event · cmp_f1ad26bc7ba688
council reasoning
B restructures the core product model (rooms vs thread tags), event schema, reducer indexing, and consolidates the scattered REST surface into a batch RPC API with CLI/tests updated to match—foundational multi-room architecture. A is a solid but narrower win: prose item-ref tokenization, stricter braced bodies, and garden linkify for URL/dash refs, plus small breadcrumb correctness.
Side A delivers cohesive, lasting parser and rendering improvements: it introduces typed deterministic block masking, a prose item-reference tokenizer that correctly avoids code fences and trims URL punctuation, enforces braced DSL bodies, updates linkification for raw URLs/external references, and adds extensive tests covering these behaviors. Side B is a very large architectural refactor that replaces many REST endpoints with a new RPC layer and renames thread/room concepts, but much of the patch is migration and code movement rather than clearly new functionality, making its enduring value less certain than A's targeted correctness improvements.
sides
A — c_11ce057e37af (tommy-mor)
message
[08565bea] Tokenize prose refs for garden URL links (#147) * Tokenize prose refs for garden URL links Co-authored-by: tommy <thmorriss@gmail.com> * Stop prose URLs at line boundaries Co-authored-by: tommy <thmorriss@gmail.com> * Require braced DSL item bodies Co-authored-by: tommy <thmorriss@gmail.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
diff preview
diff --git a/server/src/dsl.rs b/server/src/dsl.rs
index 4203c2f59dd8825d7a91513c4023f3ccd37f1efc..b3c785674560a0b42a7f4c3aa13d80cd350f485b 100644
--- a/server/src/dsl.rs
+++ b/server/src/dsl.rs
@@ -1,7 +1,5 @@
use std::collections::HashMap;
-use rand::Rng;
-
/// Parsed DSL document.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Document {
@@ -39,31 +37,57 @@ pub enum DslError {
/// Matches the legacy Python parser behavior:
/// - Supports toggle markers (open == close), e.g. ```...```
/// - Supports nested markers (open != close), e.g. { ... { ... } ... }
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum BlockKind {
+ CodeFence,
+ DoubleBrace,
+ Brace,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+struct MaskedBlock {
+ kind: BlockKind,
+}
+
#[derive(Debug, Default, Clone)]
pub struct BlockMasker {
pub replacements: HashMap<String, String>,
+ blocks: HashMap<String, MaskedBlock>,
+ next_id: u32,
}
impl BlockMasker {
pub fn new() -> Self {
Self {
replacements: HashMap::new(),
+ blocks: HashMap::new(),
+ next_id: 0,
}
}
- fn new_token(&mut self) -> String {
- let mut rng = rand::thread_rng();
- let n: u32 = rng.gen();
- let token = format!("__BLOCK_{:08x}__", n);
- // Extremely unlikely collision; if it happens, regenerate.
- if self.replacements.contains_key(&token) {
- return self.new_token();
+ fn new_token(&mut self, haystack: &str) -> String {
+ loop {
+ let token = format!("__BLOCK_{:08x}__", self.next_id);
+ self.next_id = self.next_id.wrapping_add(1);
+ if !self.replacements.contains_key(&token) && !haystack.contains(&token) {
+ return token;
+ }
}
- token
}
/// Replace outermost balanced blocks with tokens.
pub fn mask(&mut self, text: &str, open_marker: &str, close_marker: &str) -> String {
+ self.mask_kind(text, open_marker, close_marker, BlockKind::Brace)
+ }
+
+ /// Replace outermost balanced blocks with typed deterministic tokens.
+ pub fn mask_kind(
+ &mut self,
+ text: &str,
+ open_marker: &str,
+ close_marker: &str,
+ kind: BlockKind,
+ ) -> String {
if text.is_empty() {
return text.to_string();
}
@@ -97,9 +121,10 @@ impl BlockMasker {
// Found end of outermost block
let s = start_idx.max(0) as usize;
let original_block = &text[s..i];
- let token = self.new_token();
+ let token = self.new_token(text);
self.replacements
.insert(token.clone(), original_block.to_string());
+ self.blocks.insert(token.clone(), MaskedBlock { kind });
result_parts.push(token);
current_idx = i;
}
@@ -176,13 +201,22 @@ impl BlockMasker {
}
token.to_string()
}
+
+ pub fn block_kind(&self, token: &str) -> Option<BlockKind> {
+ self.blocks.get(token).map(|b| b.kind)
+ }
}
fn mask_all(mut masker: BlockMasker, text: &str) -> (BlockMasker, String) {
// Mask hierarchy: Code -> Double Brace -> Single Brace.
- let t = masker.mask(text, "```", "```");
- let t = masker.mask(&t, "{{", "}}");
- let t = masker.mask(&t, "{", "}");
+ let t = masker.mask_kind(text, "```", "```", BlockKind::CodeFence);
+ let t = masker.mask_kind(&t, "{{", "}}", BlockKind::DoubleBrace);
+ let t = masker.mask_kind(&t, "{", "}", BlockKind::Brace);
+ (masker, t)
+}
+
+fn mask_code_fences(mut masker: BlockMasker, text: &str) -> (BlockMasker, String) {
+ let t = masker.mask_kind(text, "```", "```", BlockKind::CodeFence);
(masker, t)
}
@@ -253,7 +287,34 @@ fn skip_ws(s: &str, mut i: usize) -> usize {
i
}
-fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ProseToken {
+ Text(String),
+ ItemRef(String),
+}
+
+fn trim_prose_item_ref_end(s: &str, mut end: usize) -> usize {
+ while end > 0 {
+ let Some((idx, c)) = s[..end].char_indices().next_back() else {
+ break;
+ };
+ if matches!(
+ c,
+ '.' | ',' | ';' | ':' | '!' | '?' | ')' | ']' | '}' | '"' | '\''
+ ) {
+ end = idx;
+ } else {
+ break;
+ }
+ }
+ end
+}
+
+fn parse_item_name_at_with_mode(
+ s: &str,
+ i: usize,
+ trim_trailing_punctuation: bool,
+) -> Option<(String, usize)> {
let bytes = s.as_bytes();
if i >= bytes.len() {
return None;
@@ -263,6 +324,9 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
if s[i..].starts_with("https://") || s[i..].starts_with("http://") {
let mut j = i;
while j < bytes.len() {
+ if trim_trailing_punctuation && bytes[j] == b'\n' {
+ break;
+ }
if bytes[j..].starts_with(b"__BLOCK_") || is_ws_byte(bytes[j]) {
break;
}
@@ -271,6 +335,12 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
if j <= i {
return None;
}
+ if trim_trailing_punctuation {
+ j = trim_prose_item_ref_end(s, j);
+ if j <= i {
+ return None;
+ }
+ }
return Some((s[i..j].to_string(), j));
}
@@ -296,6 +366,12 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
if j <= i + 2 {
return None;
}
+ if trim_trailing_punctuation {
+ j = trim_prose_item_ref_end(s, j);
+ if j <= i + 2 {
+ return None;
+ }
+ }
let raw = &s[i..j];
if !is_item_name(raw) {
return None;
@@ -336,6 +412,46 @@ fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
Some((format!("~/{}", name), j))
}
+fn parse_item_name_at(s: &str, i: usize) -> Option<(String, usize)> {
+ parse_item_name_at_with_mode(s, i, false)
+}
+
+pub fn parse_prose_item_ref_at(s: &str, i: usize) -> Option<(String, usize)> {
+ parse_item_name_at_with_mode(s, i, true)
+}
+
+pub fn tokenize_prose_item_refs(text: &str) -> Vec<ProseToken> {
+ if text.is_empty() {
+ return Vec::new();
+ }
+ let (masker, masked) = mask_code_fences(BlockMasker::new(), text);
+ let mut tokens = Vec::new();
+ let mut text_start = 0usize;
+ let mut i = 0usize;
+
+ while i < masked.len() {
+ if let Some((raw, end)) = parse_prose_item_ref_at(&masked, i) {
+ if text_start < i {
+ tokens.push(ProseToken::Text(masker.unmask(&masked[text_start..i])));
+ }
+ tokens.push(ProseToken::ItemRef(masker.unmask(&raw)));
+ i = end;
+ text_start = i;
+ continue;
+ }
+
+ let Some((_, c)) = masked[i..].char_indices().next() else {
+ break;
+ };
+ i += c.len_utf8();
+ }
+
+ if text_start < masked.len() {
+ tokens.push(ProseToken::Text(masker.unmask(&masked[text_start..])));
+ }
+ tokens
+}
+
fn parse_block_token_at(s: &str, i: usize) -> Option<(String, usize)> {
let bytes = s.as_bytes();
if i >= bytes.len() {
@@ -401,6 +517,12 @@ fn parse_block_prefixed_statement(
tail: &str,
masker: &BlockMasker,
) -> Result<Stmt, DslError> {
+ if masker.block_kind(block_token) == Some(BlockKind::CodeFence) {
+ return Err(DslError::Parse(
+ "vote explanations must use `{ ... }`; code fences belong inside body blocks"
+ .to_string(),
+ ));
+ }
// vote: block item_ref comparison item_ref
let s = tail.trim_start();
if s.is_empty() {
@@ -462,6 +584,11 @@ fn parse_item_definition_statement(stripped: &str, masker: &BlockMasker) -> Resu
}
if let Some((tok, end)) = parse_block_token_at(stripped, i) {
+ if masker.block_kind(&tok) == Some(BlockKind::CodeFence) {
+ return Err(DslError::Parse(
+ "item bodies must use `{ ... }`; code fences belong inside body blocks".to_string(),
+ ));
+ }
let body = masker.extract_body(&tok);
let tail = stripped[end..].trim();
if !tail.is_empty() {
@@ -572,6 +699,10 @@ pub fn parse_full(text: &str) -> Result<Document, DslError> {
if let Some((tok, end)) = parse_block_token_at(stripped, 0) {
if stripped[end..].trim().is_empty() {
+ if masker.block_kind(&tok) == Some(BlockKind::CodeFence) {
+ prose_buffer.push(line);
+ continue;
+ }
pending_block = Some(tok);
continue;
}
@@ -619,6 +750,70 @@ mod tests {
assert_eq!(roundtrip, input);
}
+ #[test]
+ fn blockmasker_tokens_are_deterministic_and_typed() {
+ let input = "x ```code``` y {body}";
+ let (masker, masked) = mask_all(BlockMasker::new(), input);
+ assert!(masked.contains("__BLOCK_00000000__"));
+ assert!(masked.contains("__BLOCK_00000001__"));
+ assert_eq!(
+ masker.block_kind("__BLOCK_00000000__"),
+ Some(BlockKind::CodeFence)
+ );
+ assert_eq!(
+ masker.block_kind("__BLOCK_00000001__"),
+ Some(BlockKind::Brace)
+ );
+ assert_eq!(masker.unmask(&masked), input);
+ }
+
+ #[test]
+ fn prose_tokenizer_finds_tilde_dash_and_raw_url_refs() {
+ let tokens =
+ tokenize_prose_item_refs("see ~/a/b then -/example.com/x and https://Example.com/A/B.");
+ assert_eq!(
+ tokens,
+ vec![
+ ProseToken::Text("see ".to_string()),
+ ProseToken::ItemRef("~/a/b".to_string()),
+ ProseToken::Text(" then ".to_string()),
+ ProseToken::ItemRef("-/example.com/x".to_string()),
+ ProseToken::Text(" and ".to_string()),
+ ProseToken::ItemRef("https://Example.com/A/B".to_string()),
+ ProseToken::Text(".".to_string()),
+ ]
+ );
+ }
+
+ #[test]
+ fn prose_tokenizer_stops_raw_urls_at_newlines() {
+ let tokens = tokenize_prose_item_refs("https://example.com/a/b.\n-/example.com/a/b");
+ assert_eq!(
+ tokens,
+ vec![
+ ProseToken::ItemRef("https://example.com/a/b".to_string()),
+ ProseToken::Text(".\n".to_string()),
+ ProseToken::ItemRef("-/example.com/a/b".to_string()),
+ ]
+ );
+ }
+
+ #[test]
+ fn prose_tokenizer_does_not_linkify_inside_code_fences() {
+ let tokens = tokenize_prose_item_refs(
+ "before ```json\n{\"url\":\"https://example.com\"}\n``` after ~/x",
+ );
+ assert_eq!(
+ tokens,
+ vec![
+ ProseToken::Text(
+ "before ```json\n{\"url\":\"https://example.com\"}\n``` after ".to_string()
+ ),
+ ProseToken::ItemRef("~/x".to_string()),
+ ]
+ );
+ }
+
#[test]
fn parse_item_with_body_strips_outer_braces() {
let input = "~/rust { Systems language }";
@@ -633,8 +828,8 @@ mod tests {
}
#[test]
- fn parse_item_with_fenced_json_body_preserves_braces() {
- let input = "~/item/in/url ```json\n{\"test\": true}\n```";
+ fn parse_item_with_braced_fenced_json_body_preserves_braces() {
+ let input = "~/item/in/url {\n```json\n{\"test\": true}\n```\n}";
let doc = parse_full(input).unwrap();
assert_eq!(
doc.statements,
@@ -645,6 +840,51 @@ mod tests {
… preview truncated; 18,752 characters omittedB — c_2595b6007624 (tommy-mor)
message
[96b6da05] rpc + reducer changes first pass
diff preview
diff --git a/cli/src/main.rs b/cli/src/main.rs
index 630c5dea1f78c0ec9bc53e6b96234a0dc75bb705..8d0442959f4332bafe499a2a8cdf364731a06871 100644
--- a/cli/src/main.rs
+++ b/cli/src/main.rs
@@ -21,8 +21,9 @@ struct Cli {
cmd: Option<Command>,
}
+/// Commands scoped to a room (`public` or `shortid/slug`).
#[derive(Subcommand, Debug)]
-enum Command {
+enum ScopedCmd {
/// Browse the garden (ontology) — light mode, ranked by votes
Garden {
#[command(subcommand)]
@@ -174,6 +175,23 @@ enum Command {
#[arg(long)]
json: bool,
},
+}
+
+#[derive(Subcommand, Debug)]
+enum Command {
+ /// Public site (same as room `public`)
+ Public {
+ #[command(subcommand)]
+ sub: ScopedCmd,
+ },
+ /// Private room id (`shortid/slug` from `room create`)
+ Private {
+ /// Room id, e.g. `a1b2c3d/my-project`
+ #[arg(value_name = "ROOM_ID")]
+ room: String,
+ #[command(subcommand)]
+ sub: ScopedCmd,
+ },
/// Show all activity since you last posted (global feed)
///
@@ -628,6 +646,37 @@ fn http_client() -> Result<reqwest::Client> {
.build()?)
}
+async fn send_rpc(
+ client: &reqwest::Client,
+ base: &str,
+ bearer: Option<&str>,
+ commands: Vec<RpcCommand>,
+) -> Result<RpcBatchResponse> {
+ let url = format!("{}/api/v0/rpc", base.trim_end_matches('/'));
+ let mut req = client.post(url).json(&RpcBatch(commands));
+ if let Some(b) = bearer {
+ req = req.header("Authorization", format!("Bearer {}", b));
+ }
+ let resp = req.send().await?;
+ let status = resp.status();
+ let text = resp.text().await.unwrap_or_default();
+ if !status.is_success() {
+ return Err(anyhow!("rpc HTTP {}: {}", status, text.trim()));
+ }
+ serde_json::from_str(&text).map_err(|e| anyhow!("rpc response: {e}"))
+}
+
+fn rpc_line_ok(line: &RpcLine) -> Result<&RpcResult> {
+ if !line.ok {
+ let mut m = line.error.clone().unwrap_or_else(|| "rpc error".into());
+ if let Some(h) = &line.hint {
+ m.push_str(&format!("\nhint: {h}"));
+ }
+ return Err(anyhow!(m));
+ }
+ line.result.as_ref().ok_or_else(|| anyhow!("rpc missing result"))
+}
+
/// Normalize ontology path for API. Accepts path with or without ~/ (shell expands ~ to $HOME).
/// Returns a bare slug path (e.g. `languages/python`) with no leading `/` or `~/`.
/// Call `ontology_path_for_api_query` before sending `item=` / `parent=` params so the server
@@ -729,231 +778,262 @@ fn write_secret_file(name: &str, contents: &str) -> Result<()> {
Ok(())
}
-#[tokio::main]
-async fn main() -> Result<()> {
- let Cli { cmd, server } = Cli::parse();
-
- // If no command provided, print the guide
- let Some(cmd) = cmd else {
- print!("{}", include_str!("../GUIDE.sorter"));
- return Ok(());
- };
-
- let base = server.trim_end_matches('/');
-
- match cmd {
- Command::Healthz { json } => {
- let client = http_client()?;
- let url = format!("{base}/healthz");
- let body = client.get(url).send().await?.text().await?;
- if json {
- // Wrap plain text response in a JSON object
- println!("{}", serde_json::json!({ "ok": true, "body": body.trim() }));
- } else {
- println!("{body}");
- }
- }
-
- Command::Search { query, json } => {
- let client = http_client()?;
- let url = format!("{base}/api/v0/search?q={}", urlencoding::encode(&query));
- let resp: slug_types::SearchResponse = expect_json(client.get(url).send().await?).await?;
- if json {
- println!("{}", serde_json::to_string_pretty(&resp)?);
- } else {
- if !resp.items.is_empty() {
- println!("items ({})", resp.items.len());
- for item in &resp.items {
- print!(" {}", item.path);
- if let Some(body) = &item.body {
- let first_line = body.lines().next().unwrap_or("").trim();
- if !first_line.is_empty() {
- print!(" {}", first_line);
+async fn run_scoped(base: &str, room: &str, sub: ScopedCmd) -> Result<()> {
+ let room = room.trim();
+ let client = http_client()?;
+ match sub {
+ ScopedCmd::Garden { sub } => match sub {
+ GardenCmd::Tree { json } => {
+ let batch = send_rpc(&client, base, None, vec![RpcCommand::GetLeaves { room: room.to_string() }]).await?;
+ match rpc_line_ok(&batch.results[0])? {
+ RpcResult::Leaves(resp) => {
+ if json {
+ println!("{}", serde_json::to_string_pretty(&resp)?);
+ } else {
+ for p in &resp.paths {
+ println!("~/{}", p);
}
}
- println!();
- }
- }
- if !resp.threads.is_empty() {
- if !resp.items.is_empty() { println!(); }
- println!("threads ({})", resp.threads.len());
- let now_ms = std::time::SystemTime::now()
- .duration_since(std::time::UNIX_EPOCH)
- .unwrap_or_default()
- .as_millis() as i64;
- for t in &resp.threads {
- println!(" {} {}n {}", t.tag, t.post_count, slug_types::timeago::timeago(now_ms, t.last_activity));
- }
- }
- if !resp.posts.is_empty() {
- if !resp.items.is_empty() || !resp.threads.is_empty() { println!(); }
- println!("posts ({})", resp.posts.len());
- let now_ms = std::time::SystemTime::now()
- .duration_since(std::time::UNIX_EPOCH)
- .unwrap_or_default()
- .as_millis() as i64;
- for p in &resp.posts {
- let first_line = p.snippet.lines().next().unwrap_or("").trim();
- println!(" {} · {} {}", p.thread, slug_types::timeago::timeago(now_ms, p.ts), first_line);
- }
- }
- if resp.items.is_empty() && resp.threads.is_empty() && resp.posts.is_empty() {
- println!("no results");
- }
- }
- }
-
- Command::Garden { sub } => match sub {
- GardenCmd::Tree { json } => {
- let client = http_client()?;
- let url = format!("{base}/api/v0/leaves");
- let builder = client.get(url);
- let resp: LeavesResponse = expect_json(builder.send().await?).await?;
- if json {
- println!("{}", serde_json::to_string_pretty(&resp)?);
- } else {
- for p in &resp.paths {
- println!("~/{}", p);
}
+ _ => return Err(anyhow!("unexpected RPC result")),
}
}
-
GardenCmd::Body { path, json, full } => {
let path = normalize_ontology_path_input(&path).map_err(anyhow::Error::msg)?;
let item_q = ontology_path_for_api_query(&path);
- let client = http_client()?;
- let mut url = format!("{base}/api/v0/item?item={}", urlencoding::encode(&item_q));
- if full {
- url.push_str("&full=true");
- }
- let builder = client.get(url);
- let resp: ItemResponse = expect_json(builder.send().await?).await?;
- if json {
- println!("{}", serde_json::to_string_pretty(&resp)?);
- } else {
- print_item_response(&resp);
+ let batch = send_rpc(
+ &client,
+ base,
+ None,
+ vec![RpcCommand::GetGardenItem {
+ room: room.to_string(),
+ item_path: item_q,
+ full: Some(full),
+ }],
+ )
+ .await?;
+ match rpc_line_ok(&batch.results[0])? {
+ RpcResult::GardenItem(resp) => {
+ if json {
+ println!("{}", serde_json::to_string_pretty(&resp)?);
+ } else {
+ print_item_response(&resp);
+ }
+ }
+ _ => return Err(anyhow!("unexpected RPC result")),
}
}
-
GardenCmd::Children { paths, depth, json } => {
let paths: Vec<String> = paths
.iter()
.map(|p| normalize_ontology_path_input(p).map_err(anyhow::Error::msg))
.collect::<Result<Vec<_>>>()?;
- let client = http_client()?;
let parent_param = paths
.iter()
.map(|p| ontology_path_for_api_query(p))
.collect::<Vec<_>>()
.join(",");
- let mut url = format!("{base}/api/v0/rank?parent={}", urlencoding::encode(&parent_param));
- if let Some(d) = depth {
- url.push_str(&format!("&depth={d}"));
- }
- let builder = client.get(url);
- let resp: RankResponse = expect_json(builder.send().await?).await?;
-
- if json {
- println!("{}", serde_json::to_string_pretty(&resp)?);
- } else {
- print_rank_response(&resp);
+ let batch = send_rpc(
+ &client,
+ base,
+ None,
+ vec![RpcCommand::GetGardenRank {
+ room: room.to_string(),
+ parent_path: parent_param,
+ depth,
+ offset: None,
+ limit: None,
+ percent: None,
+ }],
+ )
+ .await?;
+ match rpc_line_ok(&batch.results[0])? {
+ RpcResult::GardenRank(resp) => {
+ if json {
+ println!("{}", serde_json::to_string_pretty(&resp)?);
+ } else {
+ print_rank_response(&resp);
+ }
+ }
+ _ => return Err(anyhow!("unexpected RPC result")),
}
}
-
GardenCmd::Pair { path, json } => {
let path = normalize_ontology_path_input(&path).map_err(anyhow::Error::msg)?;
let parent_q = ontology_path_for_api_query(&path);
- let client = http_client()?;
- let url = format!("{base}/api/v0/pair?parent={}", urlencoding::encode(&parent_q));
- let builder = client.get(url);
- let resp: PairResponse = expect_json(builder.send().await?).await?;
- if json {
- println!("{}", serde_json::to_string_pretty(&resp)?);
- } else {
- print_pair_response(&resp);
+ let batch = send_rpc(
+ &client,
+ base,
+ None,
+ vec![RpcCommand::GetPair {
+ room: room.to_string(),
+ parent_path: parent_q,
+ }]
… preview truncated; 237,234 characters omittedHardlinks — judgments / attempts / prompt
judgments
attempts
Prompt text is loaded only by the download route.