Side B implements a substantial, self-contained new feature: an async Reddit fetch broker with OAuth handling, rate-limit backoff, caching, JSON parsing for subreddits/posts, and tests, wired cleanly into AppState and the browse handler. Side A is a large, mostly mechanical DSL syntax reshuffle (trailing to leading explanation blocks) touching many tests/fixtures/docs, which is a real but narrower, more churn-heavy change with less net new capability despite its size.
constitution · epochs · watch · epoch 3
comparison
c_2f5d9e0370f8 (tommy-mor) vs c_c124c217f89c (tommy-mor)
download prompt · raw event · cmp_909d61d72ee372
council reasoning
A redesigns the core sorter DSL (block/explanation-first votes, clearer item bodies) with real parser changes in dsl.rs plus coordinated docs/UI/fixture updates, which shapes how every ingest works. B adds a solid Reddit fetch worker (OAuth, rate limits, URL mapping) but it is a peripheral, still-incomplete integration on a placeholder path rather than a lasting core-language change.
Side A implements a substantive DSL redesign by changing vote syntax to use leading explanation blocks, refactoring the parser to support pending block parsing, adding validation and error cases, updating UI generation, and comprehensively updating tests and documentation to match the new grammar. Side B adds a useful but incomplete Reddit background fetch pipeline and request hooks, but much of it is new integration code without equivalent end-to-end validation and includes placeholder planning, making its lasting project impact smaller than the coordinated language and parser change in Side A.
sides
A — c_2f5d9e0370f8 (tommy-mor)
message
[6bda2635] Use title-first items and explanation-first votes (#135) * Require block-first sorter DSL statements Co-authored-by: tommy <thmorriss@gmail.com> * Use title-first items with explanation-first votes Co-authored-by: tommy <thmorriss@gmail.com> * Update garden vote test DSL fixtures Co-authored-by: tommy <thmorriss@gmail.com> * Update browser vote DSL payloads Co-authored-by: tommy <thmorriss@gmail.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
diff preview
diff --git a/cli/DSL.txt b/cli/DSL.txt
index 18f69ef25f01583fddf9fc90e077bb2c3fa72bb6..c9b12bf0edaf73ad252f0a202b7fe61e311df50a 100644
--- a/cli/DSL.txt
+++ b/cli/DSL.txt
@@ -18,9 +18,20 @@ Blank lines are preserved to maintain paragraph structure.
~/item/a {itembody}
~/python { A high-level scripting language }
-~/item/a > ~/python { Item A is better because of X. }
-~/python 3:1 ~/go { Python's ecosystem is much richer than Go's. }
-https://example.com/lang = ~/go { They are equally good in this context. }
+{
+Item A is better because of X.
+}
+~/item/a > ~/python
+
+{
+Python's ecosystem is much richer than Go's.
+}
+~/python 3:1 ~/go
+
+{
+They are equally good in this context.
+}
+https://example.com/lang = ~/go
```
SYNTAX RULES
@@ -35,7 +46,7 @@ Starts a thread. Tag allows alphanumeric, `-`, `_`, and `/`. Subtitle max 100 ch
~/<local-item-path> { description }
```
Defines an ontology item (garden layer). Paths can be nested (e.g. `~/languages/python`).
-A leading `/` alone is **not** allowed in the DSL — use `~/` only. Descriptions (bodies) are wrapped in `{}`. Can be adjacent (e.g. `~/arrived{ready}`).
+A leading `/` alone is **not** allowed in the DSL — use `~/` only. Descriptions (bodies) are wrapped in `{}` and follow the item path.
```sorter
https://example.com/item { description }
@@ -46,7 +57,10 @@ Canonicalization rules for URLs:
- `~/` and `https://slug.social/~/` map to the same local item path.
```sorter
-<item1> <comparison> <item2> { required explanation }
+{
+required explanation
+}
+<item1> <comparison> <item2>
```
Compares two items. The explanation is REQUIRED.
Comparisons:
@@ -65,7 +79,8 @@ When writing bodies or explanations, you can use braces `{}` and code blocks wit
3. Single braces: { ... }
```sorter
-~/code { Here is a block: ```def foo(): return {"a": 1}``` }
+{ Here is a block: ```def foo(): return {"a": 1}``` }
+~/code
```
STYLE
diff --git a/cli/GUIDE.sorter b/cli/GUIDE.sorter
index 7ea1c2649cb4a323b46f0bf389025079a9c9ab43..86accba72ecea547215d947fb6552e0ead25687c 100644
--- a/cli/GUIDE.sorter
+++ b/cli/GUIDE.sorter
@@ -83,7 +83,8 @@ Item definitions (attaches a description to an item):
~/thread/item { description }
Comparisons:
- ~/thread/item-a 3:1 ~/thread/item-b { reasoning }
+ { reasoning }
+ ~/thread/item-a 3:1 ~/thread/item-b
Ratio formats:
3:1 left is 3x better than right
@@ -95,8 +96,7 @@ Shorthand:
< means 1:2 (right is better)
= means 1:1 (equal)
-Bodies can attach without whitespace:
- ~/thread/item{Description here}
+Item bodies follow the item path. Vote explanations come first; the comparison is the verdict line.
}
You can write any prose in your posts. These won't be part of the garden but only the thread.
@@ -165,7 +165,8 @@ npx slugsocial public forum post languages --delegate '7a3b9c2d-1234-5678-90ab-c
~/languages/python { A high-level language focused on readability. }
~/languages/rust { A systems language focused on safety and performance. }
-~/languages/python 2:1 ~/languages/rust { Python has simpler syntax for beginners - fewer symbols, explicit over implicit. Rust's borrow checker adds cognitive load even for simple programs. Both are readable once learned, but Python's learning curve is gentler. }
+{ Python has simpler syntax for beginners - fewer symbols, explicit over implicit. Rust's borrow checker adds cognitive load even for simple programs. Both are readable once learned, but Python's learning curve is gentler. }
+~/languages/python 2:1 ~/languages/rust
EOF
# See current ranking
diff --git a/ideas/single-thread.md b/ideas/single-thread.md
index 86230fe6119c31045c21dbfcb12311b323c02fa8..0efb7199524cc3b308b1922c03e3a99193e4586c 100644
--- a/ideas/single-thread.md
+++ b/ideas/single-thread.md
@@ -15,7 +15,8 @@ Previously a `.sorter` document could scatter `#tags` throughout:
~/languages/rust {A systems language.}
#tools
~/tools/cargo {Rust's build system.}
-~/languages/rust 2:1 ~/tools/cargo {Rust is more foundational than its tooling.}
+{Rust is more foundational than its tooling.}
+~/languages/rust 2:1 ~/tools/cargo
```
The system would fan the ingest into both `#languages` and `#tools` — the same
diff --git a/server/src/api/mod.rs b/server/src/api/mod.rs
index 9f3c1cc21c2ae157c024a447980a97e190c8f066..920e967b47852ea82fa61b84c457ae3582dd9800 100644
--- a/server/src/api/mod.rs
+++ b/server/src/api/mod.rs
@@ -72,16 +72,16 @@ mod tests {
apply_ingest(
&mut reduced,
1,
- "~/t/a {a}\n~/t/b {b}\n~/t/a 2:1 ~/t/b {because}\n",
+ "~/t/a {a}\n~/t/b {b}\n{because}\n~/t/a 2:1 ~/t/b\n",
);
- let text = "~/t/a 1:1 ~/t/b {equal}\n";
+ let text = "{equal}\n~/t/a 1:1 ~/t/b\n";
validate_ingest_document(&reduced, text, &crate::reducer::ScopeId::Public).unwrap();
}
#[test]
fn validate_ingest_document_rejects_vote_on_undefined_item() {
let reduced = ReducerState::default();
- let text = "~/t/a {x}\n~/t/b 1:1 ~/t/missing {why}\n";
+ let text = "~/t/a {x}\n{why}\n~/t/b 1:1 ~/t/missing\n";
let err = validate_ingest_document(&reduced, text, &crate::reducer::ScopeId::Public).unwrap_err();
assert_eq!(err.0, StatusCode::BAD_REQUEST);
assert!(err.1.contains("undefined item"));
diff --git a/server/src/api/ui_html.rs b/server/src/api/ui_html.rs
index 696e7b3605e2e68aee0351116c494c2958506add..7cbda3876451687aa7a55547fc06bbe86ac9d260 100644
--- a/server/src/api/ui_html.rs
+++ b/server/src/api/ui_html.rs
@@ -222,13 +222,13 @@ async fn dispatch_ui_action(
}
let text = format!(
- "@{}\n{} {}:{} {} {{\n{}\n}}\n",
+ "@{}\n{{\n{}\n}}\n{} {}:{} {}\n",
crate::api::auth::WEB_BROWSER_AGENT,
+ exp,
left_id.as_str(),
rl,
rr,
- right_id.as_str(),
- exp
+ right_id.as_str()
);
match rpc_post_with_bearer(state, &session.bearer, room.clone(), thread_tag.clone(), text).await {
diff --git a/server/src/dsl.rs b/server/src/dsl.rs
index 7962342bd023b3b5061a684d84d4ab164134b111..def8b497c71567016a522648b9630d7038163e41 100644
--- a/server/src/dsl.rs
+++ b/server/src/dsl.rs
@@ -11,16 +11,21 @@ pub struct Document {
/// A single statement in the DSL (or prose when using `parse_full`).
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Stmt {
- Item { title: String, body: Option<String> },
+ Item {
+ title: String,
+ body: Option<String>,
+ },
Vote {
item1: String,
item2: String,
ratio_left: i32,
ratio_right: i32,
- /// Required non-empty explanation (from trailing `{ ... }`).
+ /// Required non-empty explanation (from leading `{ ... }`).
explanation: String,
},
- Prose { text: String },
+ Prose {
+ text: String,
+ },
}
#[derive(Debug, thiserror::Error)]
@@ -391,71 +396,46 @@ fn parse_comparison_at(s: &str, i: usize) -> Option<((i32, i32), usize)> {
Some(((left, right), j))
}
-fn parse_item_statement(stripped: &str, masker: &BlockMasker) -> Result<Stmt, DslError> {
- // item: ("~/" | "https://..." | "http://...") item_ref body?
- // vote: same for both operands.
- //
- // Important: body token can be adjacent to the item name (no whitespace),
- // e.g. "~/arrived{...}" -> "~/arrived__BLOCK_x__".
- let s = stripped;
- let bytes = s.as_bytes();
- if bytes.is_empty() {
- return Err(DslError::Parse("missing item statement".to_string()));
+fn parse_block_prefixed_statement(
+ block_token: &str,
+ tail: &str,
+ masker: &BlockMasker,
+) -> Result<Stmt, DslError> {
+ // vote: block item_ref comparison item_ref
+ let s = tail.trim_start();
+ if s.is_empty() {
+ return Err(DslError::Parse(
+ "missing vote statement after leading explanation block".to_string(),
+ ));
}
let (item1, j) =
parse_item_name_at(s, 0).ok_or_else(|| DslError::Parse("invalid item name".to_string()))?;
+ let explanation = masker.extract_body(block_token);
+ let mut i = skip_ws(s, j);
- // Either we have:
- // - immediate/whitespace block token => Item
- // - comparison => Vote
- // - whitespace then block token => Item
- // - whitespace then comparison => Vote
- let i = skip_ws(s, j);
-
- // If next is end or a block token => Item.
if i >= s.len() {
- return Ok(Stmt::Item {
- title: item1,
- body: None,
- });
- }
- if let Some((tok, end)) = parse_block_token_at(s, i) {
- let body = masker.extract_body(&tok);
- let tail = s[end..].trim();
- if !tail.is_empty() {
- return Err(DslError::Parse("extra tokens after item".to_string()));
- }
- return Ok(Stmt::Item {
- title: item1,
- body: Some(body),
- });
+ return Err(DslError::Parse(
+ "leading `{ ... }` blocks are vote explanations; item bodies belong after item paths"
+ .to_string(),
+ ));
}
- // Otherwise parse comparison then "/item2" then REQUIRED body.
- let ((ratio_left, ratio_right), mut k) = parse_comparison_at(s, i)
+ let ((ratio_left, ratio_right), k) = parse_comparison_at(s, i)
.ok_or_else(|| DslError::Parse(format!("invalid comparison near: {}", &s[i..])))?;
if ratio_left == 0 && ratio_right == 0 {
return Err(DslError::Parse(
"vote ratio 0:0 is invalid; use 1:1 for a tie or omit the vote".to_string(),
));
}
- k = skip_ws(s, k);
- let (item2, mut m) = parse_item_name_at(s, k)
+ i = skip_ws(s, k);
+ let (item2, m) = parse_item_name_at(s, i)
.ok_or_else(|| DslError::Parse("invalid rhs item name".to_string()))?;
- m = skip_ws(s, m);
-
- let Some((tok, end)) = parse_block_token_at(s, m) else {
- return Err(DslError::Parse(
- "missing vote explanation (add a trailing `{ ... }`)".to_string(),
- ));
- };
- let explanation = masker.extract_body(&tok);
+ i = skip_ws(s, m);
if explanation.trim().is_empty() {
return Err(DslError::Parse("empty vote explanation".to_string()));
}
- m = end;
- let tail = s[m..].trim();
+ let tail = s[i..].trim();
if !tail.is_empty() {
return Err(DslError::Parse("extra tokens after vote".to_string()));
}
@@ -469,6 +449,35 @@ fn parse_item_statement(stripped: &str, masker: &BlockMasker) -> Result<Stmt, Ds
})
}
+fn parse_item_definition_statement(stripped: &str, masker: &BlockMasker) -> Result<Stmt, DslError> {
+ let (item1, j) =
+ parse_item_name_at(stripped, 0).ok_or_else(|| DslError::Parse("invalid item name".to_string()))?;
+ let i = skip_ws(stripped, j);
+
+ if i >= stripped.len() {
+ return Ok(Stmt::Item {
+ title: item1,
+ body: None,
+ });
+ }
+
+ if let Some((tok, end)) = parse_block_token_at(stripped, i) {
+ let body = masker.extract_body(&tok);
+ let tail = stripped[end..].trim();
+ if !tail.is_empty() {
+ return Err(DslError::Parse("extra tokens after item".to_string()));
+ }
+ return Ok(Stmt::Item {
+ title: item1,
+ body: Some(body),
+ });
+ }
+
+ Err(DslError::Parse(
+ "vote explanations must start with a `{ ... }` block before the comparison".to_string(),
+ ))
+}
+
fn parse_line(masked_line: &str, masker: &BlockMasker) -> Result<Vec<Stmt>, DslError> {
let stripped = masked_line.trim_start();
if stripped.is_empty() {
@@ -477,27 +486,28 @@ fn parse_line(masked_line: &str, masker: &BlockMasker) -> Result<Vec<Stmt>, DslE
let first = stripped.chars().next().unwrap();
match first {
'#' => Err(DslError::Parse("not a DS
… preview truncated; 47,409 characters omittedB — c_c124c217f89c (tommy-mor)
message
[8d8230d1] reddit
diff preview
diff --git a/.gitignore b/.gitignore
index 4c7073f9fac0c30fd2050d79a60ef447af58ebeb..ada462e900d24a3a6d08165d158c80f79c35a5a5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,3 +7,4 @@
data/
repomix-output.xml
dev-data/
+.env
diff --git a/server/Cargo.toml b/server/Cargo.toml
index 7906a8547d56b8e6a48ef59c37aa82a8510fdee9..4677fedcb45292eebebe7e9cf6ce2f5738f18ddf 100644
--- a/server/Cargo.toml
+++ b/server/Cargo.toml
@@ -16,6 +16,7 @@ tower = "0.5"
tower-http = { version = "0.5", features = ["trace"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
+reqwest = { version = "0.12", features = ["json"] }
[dev-dependencies]
reqwest = { version = "0.12", features = ["json"] }
diff --git a/server/src/html/mod.rs b/server/src/html/mod.rs
index 2864407ed6e8ec284a1dc663acf1805534566b62..df6505021d9f446c2b453e20e3eb3cf696a111f9 100644
--- a/server/src/html/mod.rs
+++ b/server/src/html/mod.rs
@@ -272,5 +272,16 @@ pub async fn home(State(state): State<AppState>, uri: Uri) -> impl IntoResponse
pub async fn browse(State(state): State<AppState>, uri: Uri) -> impl IntoResponse {
let item = ItemId::from_browse_uri(uri.path()).unwrap_or(ItemId::root());
+ if item.as_str().starts_with("reddit.com") {
+ let needs_fetch = {
+ let tree = state.tree.read().await;
+ tree.get(&item)
+ .map(|n| n.data.is_none())
+ .unwrap_or(true)
+ };
+ if needs_fetch {
+ state.reddit.request_fetch(item.clone());
+ }
+ }
item_page(state, uri, item).await
}
diff --git a/server/src/reddit.rs b/server/src/reddit.rs
index d203dca09245daf869b3aa942898447700ae69fb..90053ad03b1d7c8e94f325dd4ee64c2b4f7da900 100644
--- a/server/src/reddit.rs
+++ b/server/src/reddit.rs
@@ -1,4 +1,12 @@
-//! Reddit API import (async, decoupled from UI request path).
+//! Reddit API import via a single background worker (rate limits, dedup, backoff).
+
+use std::collections::{HashMap, HashSet};
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+
+use reqwest::{header, Client, StatusCode};
+use serde::Deserialize;
+use tokio::sync::{mpsc, RwLock};
use crate::{
path_types::ItemId,
@@ -10,12 +18,401 @@ pub fn ensure_partial_tree(tree: &mut GlobalTree, id: &ItemId) {
tree.ensure_path(id);
}
-/// Placeholder for Reddit JSON import. Returns entity data when implemented.
-pub async fn fetch_reddit_entity(_id: &ItemId) -> Option<EntityData> {
- None
+pub struct RedditCommand {
+ pub id: ItemId,
+}
+
+#[derive(Clone)]
+pub struct RedditBroker {
+ tx: mpsc::Sender<RedditCommand>,
+}
+
+#[derive(Clone)]
+struct RedditCredentials {
+ client_id: String,
+ client_secret: String,
+}
+
+struct OAuthToken {
+ access_token: String,
+ expires_at: Instant,
+}
+
+impl RedditBroker {
+ pub fn spawn(tree: Arc<RwLock<GlobalTree>>, user_agent: &str) -> Self {
+ let (tx, rx) = mpsc::channel(100);
+
+ let mut headers = header::HeaderMap::new();
+ headers.insert(
+ header::USER_AGENT,
+ header::HeaderValue::from_str(user_agent).expect("valid user agent"),
+ );
+
+ let client = Client::builder()
+ .default_headers(headers)
+ .timeout(Duration::from_secs(15))
+ .build()
+ .expect("reqwest client");
+
+ let creds = RedditCredentials::from_env();
+ tokio::spawn(reddit_worker(rx, tree, client, creds));
+
+ Self { tx }
+ }
+
+ /// Fire-and-forget: queue a fetch; worker updates the tree when done.
+ pub fn request_fetch(&self, id: ItemId) {
+ let _ = self.tx.try_send(RedditCommand { id });
+ }
+}
+
+impl RedditCredentials {
+ fn from_env() -> Option<Self> {
+ let client_id = std::env::var("REDDIT_CLIENT_ID").ok()?;
+ let client_secret = std::env::var("REDDIT_CLIENT_SECRET").ok()?;
+ if client_id.is_empty() || client_secret.is_empty() {
+ return None;
+ }
+ Some(Self {
+ client_id,
+ client_secret,
+ })
+ }
+}
+
+pub fn default_user_agent() -> String {
+ std::env::var("REDDIT_USER_AGENT").unwrap_or_else(|_| {
+ "web:sorter2.social:v0.0.1 (by /u/sorter2)".to_string()
+ })
}
-/// Apply fetched entity data to a node (called from async worker).
-pub fn apply_entity(tree: &mut GlobalTree, id: &ItemId, data: EntityData) {
- tree.set_entity_data(id, data);
+async fn reddit_worker(
+ mut rx: mpsc::Receiver<RedditCommand>,
+ tree: Arc<RwLock<GlobalTree>>,
+ client: Client,
+ creds: Option<RedditCredentials>,
+) {
+ let mut in_flight = HashSet::new();
+ let mut recently_fetched: HashMap<ItemId, Instant> = HashMap::new();
+ let mut current_delay = Duration::from_secs(1);
+ let mut oauth: Option<OAuthToken> = None;
+ let cache_ttl = Duration::from_secs(300);
+
+ while let Some(cmd) = rx.recv().await {
+ let now = Instant::now();
+ recently_fetched.retain(|_, t| now.duration_since(*t) < cache_ttl);
+
+ if in_flight.contains(&cmd.id) || recently_fetched.contains_key(&cmd.id) {
+ continue;
+ }
+
+ in_flight.insert(cmd.id.clone());
+ let fetch_id = cmd.id.clone();
+
+ tokio::time::sleep(current_delay).await;
+
+ if let Some(c) = &creds {
+ oauth = ensure_oauth_token(&client, c, oauth.take()).await;
+ }
+
+ let token = oauth.as_ref().map(|t| t.access_token.as_str());
+ let use_oauth = token.is_some();
+
+ match do_fetch(&client, &fetch_id, use_oauth, token).await {
+ Ok(FetchOutcome::Entity(data)) => {
+ let mut w = tree.write().await;
+ w.set_entity_data(&fetch_id, data);
+ recently_fetched.insert(fetch_id.clone(), Instant::now());
+ current_delay = Duration::from_millis(600);
+ }
+ Ok(FetchOutcome::NotFound) => {
+ recently_fetched.insert(fetch_id.clone(), Instant::now());
+ }
+ Ok(FetchOutcome::RateLimited { reset_secs }) => {
+ let wait = Duration::from_secs(reset_secs.max(1));
+ tracing::warn!(
+ "Reddit rate limit for {}; sleeping {}s",
+ fetch_id,
+ wait.as_secs()
+ );
+ tokio::time::sleep(wait).await;
+ current_delay = (current_delay * 2).min(Duration::from_secs(60));
+ }
+ Err(e) => {
+ tracing::warn!("Reddit fetch failed for {}: {}", fetch_id, e);
+ current_delay = (current_delay * 2).min(Duration::from_secs(60));
+ }
+ }
+
+ in_flight.remove(&fetch_id);
+ }
+}
+
+enum FetchOutcome {
+ Entity(EntityData),
+ NotFound,
+ RateLimited { reset_secs: u64 },
+}
+
+async fn ensure_oauth_token(
+ client: &Client,
+ creds: &RedditCredentials,
+ existing: Option<OAuthToken>,
+) -> Option<OAuthToken> {
+ if let Some(t) = existing {
+ if Instant::now() < t.expires_at - Duration::from_secs(60) {
+ return Some(t);
+ }
+ }
+
+ let resp = client
+ .post("https://www.reddit.com/api/v1/access_token")
+ .basic_auth(&creds.client_id, Some(&creds.client_secret))
+ .form(&[("grant_type", "client_credentials")])
+ .send()
+ .await;
+
+ let resp = match resp {
+ Ok(r) => r,
+ Err(e) => {
+ tracing::warn!("Reddit OAuth token request failed: {e}");
+ return None;
+ }
+ };
+
+ if !resp.status().is_success() {
+ tracing::warn!("Reddit OAuth token HTTP {}", resp.status());
+ return None;
+ }
+
+ #[derive(Deserialize)]
+ struct TokenResponse {
+ access_token: String,
+ expires_in: u64,
+ }
+
+ let body: TokenResponse = match resp.json().await {
+ Ok(b) => b,
+ Err(e) => {
+ tracing::warn!("Reddit OAuth token parse failed: {e}");
+ return None;
+ }
+ };
+
+ Some(OAuthToken {
+ access_token: body.access_token,
+ expires_at: Instant::now() + Duration::from_secs(body.expires_in),
+ })
+}
+
+async fn do_fetch(
+ client: &Client,
+ id: &ItemId,
+ use_oauth: bool,
+ bearer: Option<&str>,
+) -> Result<FetchOutcome, String> {
+ let url = map_item_to_reddit_api(id, use_oauth);
+ if url.is_empty() {
+ return Ok(FetchOutcome::NotFound);
+ }
+
+ let mut req = client.get(&url);
+ if let Some(token) = bearer {
+ req = req.bearer_auth(token);
+ }
+
+ let resp = req.send().await.map_err(|e| e.to_string())?;
+
+ if resp.status() == StatusCode::TOO_MANY_REQUESTS {
+ let reset = rate_limit_reset_secs(&resp);
+ return Ok(FetchOutcome::RateLimited { reset_secs: reset });
+ }
+
+ if resp.status() == StatusCode::SERVICE_UNAVAILABLE {
+ return Err("Reddit unavailable (503)".to_string());
+ }
+
+ if !resp.status().is_success() {
+ return Ok(FetchOutcome::NotFound);
+ }
+
+ if rate_limit_remaining(&resp) == Some(0) {
+ let reset = rate_limit_reset_secs(&resp);
+ return Ok(FetchOutcome::RateLimited { reset_secs: reset });
+ }
+
+ let bytes = resp.bytes().await.map_err(|e| e.to_string())?;
+ Ok(parse_reddit_json(id, &bytes)
+ .map(FetchOutcome::Entity)
+ .unwrap_or(FetchOutcome::NotFound))
+}
+
+fn rate_limit_remaining(resp: &reqwest::Response) -> Option<u64> {
+ resp.headers()
+ .get("x-ratelimit-remaining")
+ .and_then(|v| v.to_str().ok())
+ .and_then(|s| s.parse::<f64>().ok())
+ .map(|f| f.floor() as u64)
+}
+
+fn rate_limit_reset_secs(resp: &reqwest::Response) -> u64 {
+ resp.headers()
+ .get("x-ratelimit-reset")
+ .and_then(|v| v.to_str().ok())
+ .and_then(|s| s.parse::<f64>().ok())
+ .map(|f| f.ceil() as u64)
+ .unwrap_or(5)
+}
+
+/// Map canonical item id to Reddit JSON API URL.
+pub fn map_item_to_reddit_api(id: &ItemId, oauth: bool) -> String {
+ let path = id.as_str();
+ if !path.starts_with("reddit.com/") && path != "reddit.com" {
+ return String::new();
+ }
+
+ let base = if oauth {
+ "https://oauth.reddit.com"
+ } else {
+ "https://www.reddit.com"
+ };
+
+ let segments: Vec<&str> = path.split('/').collect();
+
+ if let Some(i) = segments.iter().position(|&p| p == "comments") {
+ if segments.len() > i + 1 {
+ let api_path = segments[1..=i + 1].join("/");
+ return format!("{base}/{api_path}.json?raw_json=1");
+ }
+ }
+
+ if segments.len() == 3 && segments[1] == "r" {
+ return format!("{base}/r/{}/about.json?raw_json=1", segments[2]);
+ }
+
+ String::new()
+}
+
+fn parse_reddit_json(id: &ItemId, bytes: &[u8]) -> Option<EntityData> {
+ let v: serde_json::Value = serde_json::from_slice(bytes).ok()?;
+ let segments: Vec<&str> = id.as_str().split('/').collect();
+
+ if segments.iter().any(|&p| p == "comments") {
+ parse_post_listing(&v)
+ } else {
+ parse_subreddit_about(&v)
+ }
+}
+
+fn parse_subreddit_about(v: &serde_json::Value) -> Option<EntityData> {
+ let data = v.get("data")?;
+ let title = data
+ .get("title")
+ .or_else(|| data.get("display_name"))
+ .and_then(|t| t.as_str())?
+ .to_string();
+ let body_html = data
+ .get("public_description_html")
+ .or_else(|| data.get("public_description"))
+ .and_then(|t| t.as_str())
+ .map(|s| s.to_string());
+ let thumb_url = data
+ .get("icon_img")
+ .or_else(|| data.get("community_icon"))
+ .and_then(|t| t.as_str())
+ .filter(|s| !s.is_empty())
+ .map(|s| s.to_string());
+
+ Some(EntityData {
+ title,
+ author: None,
+ body_html,
+ thumb_url,
+ })
+}
+
+fn parse_post_listing(v:
… preview truncated; 4,570 characters omittedHardlinks — judgments / attempts / prompt
judgments
attempts
Prompt text is loaded only by the download route.