B implements a real, self-contained feature: a background Reddit fetch worker with OAuth token handling, rate-limit/backoff logic, JSON parsing, and unit tests, wired cleanly into AppState—genuine new capability with lasting value. A is a URL-refactor that centralizes room-route encoding (good) but changes the public room URL format from '/r/{short}/{slug}' to '/r/{short}{slug}' (ambiguous, harder to read) and bundles in a large, seemingly unrelated speculative youtube-normalization/url-identity-test module that adds significant surface area without clear necessity for the stated 'url refactor' goal.
constitution · epochs · watch · epoch 3
comparison
c_45dfa34511fe (tommy-mor) vs c_c124c217f89c (tommy-mor)
download prompt · raw event · cmp_aec0ac26446ebe
council reasoning
A lands durable core design: centralized room path encoding (`room_route_segment` / fixed-width short id), route and link call sites rewritten to one segment, plus real external URL identity normalization (YouTube rewrites, sorted query pairs) with tests—replacing scattered `split_once('/')` URL building. B is a solid vertical feature (Reddit broker, OAuth, rate-limit/dedup worker, parsers) that replaces a stub, but it is product-specific integration plus a scratch `todo` file, so it adds less lasting cross-cutting value than A’s URL/types foundation.
Side A performs a broad but coherent refactor that centralizes room URL encoding/decoding into shared helpers (`room_route_segment`, `room_id_from_route_segment`, `ROOM_SHORT_ID_LEN`), updates routing and link generation consistently across the server, and adds URL normalization logic with tests, reducing duplicated parsing and making URL handling more maintainable. Side B adds a substantial Reddit background fetch pipeline with OAuth, rate limiting, and parsing, but it is a new feature with placeholders still remaining (for example the TODO file and no rendering changes), making its long-term project impact less certain than A's foundational cleanup and normalization.
sides
A — c_45dfa34511fe (tommy-mor)
message
[5ca518f6] url refactor
diff preview
diff --git a/Cargo.lock b/Cargo.lock
index 67a09a3b54f778fa7e857fdd589c3ed9c92e1322..ad7e4fe6d4ba2f2b033916194c1ef1ed873f1d46 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -1757,6 +1757,7 @@ name = "slug-types"
version = "0.1.0"
dependencies = [
"serde",
+ "url",
]
[[package]]
@@ -2272,6 +2273,7 @@ dependencies = [
"idna",
"percent-encoding",
"serde",
+ "serde_derive",
]
[[package]]
diff --git a/server/src/api/ui_html.rs b/server/src/api/ui_html.rs
index 606f7d6f97a4428efb90d1e0d544934c861fc4c5..cd3e0f0afd972d9ad9e7e4b92c5fa4c22bb8f620 100644
--- a/server/src/api/ui_html.rs
+++ b/server/src/api/ui_html.rs
@@ -270,10 +270,10 @@ fn post_redirect_location(room: &str, thread_tag: &str) -> String {
format!("/t/{tag}")
} else {
let room = room.trim();
- let Some((a, b)) = room.split_once('/') else {
+ let Some(seg) = slug_types::room_route_segment(room) else {
return "/".to_string();
};
- format!("/r/{a}/{b}/t/{tag}")
+ format!("/r/{seg}/t/{tag}")
}
}
diff --git a/server/src/api/write_actor.rs b/server/src/api/write_actor.rs
index cb78d2f3f95b1bc163c1fb064d1d5f657e18000f..f9c3b8bd3fbf8fcb9c035e1a1572fef0b08fa8a9 100644
--- a/server/src/api/write_actor.rs
+++ b/server/src/api/write_actor.rs
@@ -19,13 +19,15 @@ use crate::{
use super::auth::{issue_token_for_user, verify_token};
use super::helpers::{now_ms, resolve_item};
use super::validate::{normalize_room_and_thread, validate_ingest_document};
-use slug_types::RpcResult;
+use slug_types::{room_route_segment, RpcResult, ROOM_SHORT_ID_LEN};
fn gen_short_id() -> String {
use rand::Rng;
const ALPHABET: &[u8] = b"0123456789abcdefghijklmnopqrstuvwxyz";
let mut rng = rand::thread_rng();
- (0..7).map(|_| ALPHABET[rng.gen_range(0..ALPHABET.len())] as char).collect()
+ (0..ROOM_SHORT_ID_LEN)
+ .map(|_| ALPHABET[rng.gen_range(0..ALPHABET.len())] as char)
+ .collect()
}
fn parse_capability(s: &str) -> Result<crate::events::ThreadCapability, String> {
@@ -55,8 +57,8 @@ async fn broadcast_web_refresh(state: &AppState, room_key: &str, thread_id: &str
let feed_id = if room_key == "public" { "thread-feed" } else { "room-thread-feed" };
let thread_url = if room_key == "public" {
format!("/t/{thread_id}")
- } else if let Some((short, slug)) = room_key.split_once('/') {
- format!("/r/{short}/{slug}/t/{thread_id}")
+ } else if let Some(seg) = room_route_segment(room_key) {
+ format!("/r/{seg}/t/{thread_id}")
} else {
format!("/t/{thread_id}")
};
@@ -78,8 +80,8 @@ async fn broadcast_web_refresh(state: &AppState, room_key: &str, thread_id: &str
let js = builder.build();
let mut path_prefixes = vec![if room_key == "public" {
"/".to_string()
- } else if let Some((short, slug)) = room_key.split_once('/') {
- format!("/r/{short}/{slug}")
+ } else if let Some(seg) = room_route_segment(room_key) {
+ format!("/r/{seg}")
} else {
"/".to_string()
}];
diff --git a/server/src/html/forum/nav.rs b/server/src/html/forum/nav.rs
index 0ee33d91160fc5542817b5e3e9ab4fee1d0e600f..48fe11e46731670874ff8b6b05baa6f09ae0b7e4 100644
--- a/server/src/html/forum/nav.rs
+++ b/server/src/html/forum/nav.rs
@@ -1,7 +1,8 @@
use crate::canonical_path::canonicalize_item;
use crate::reducer::ScopeId;
+use slug_types::room_route_segment;
-/// URL helpers for public `/t/…` and private room threads `/r/{short}/{slug}/t/…`.
+/// URL helpers for public `/t/…` and private room threads `/r/{short}{slug}/t/…`.
#[derive(Clone)]
pub struct ThreadNav {
pub room_wire: String,
@@ -22,18 +23,15 @@ impl ThreadNav {
}
}
- /// `room_id` wire form `shortid/slug`.
+ /// `room_id` wire form `shortid/slug` (HTTP uses [`slug_types::room_route_segment`]).
pub(crate) fn from_room_id(room_id: &str) -> Option<Self> {
- let (short, slug) = room_id.split_once('/')?;
- if short.is_empty() || slug.is_empty() {
- return None;
- }
+ let room_seg = room_route_segment(room_id)?;
Some(Self {
room_wire: room_id.to_string(),
scope: ScopeId::Room(room_id.to_string()),
- room_path: format!("/r/{short}/{slug}"),
- thread_path_prefix: format!("/r/{short}/{slug}/t"),
- garden_path_prefix: format!("/r/{short}/{slug}/~"),
+ room_path: format!("/r/{room_seg}"),
+ thread_path_prefix: format!("/r/{room_seg}/t"),
+ garden_path_prefix: format!("/r/{room_seg}/~"),
})
}
diff --git a/server/src/html/forum/post_single.rs b/server/src/html/forum/post_single.rs
index c316f8f836df9d4ef9c05ebd9e54f699540e6d72..473747b3da3d4d7a54b5e0c63165d2533df643e1 100644
--- a/server/src/html/forum/post_single.rs
+++ b/server/src/html/forum/post_single.rs
@@ -93,12 +93,14 @@ pub async fn thread_post_view(
pub async fn room_thread_post_view(
State(state): State<AppState>,
- Path((room_short, room_slug, tag, index_str)): Path<(String, String, String, String)>,
+ Path((room_key, tag, index_str)): Path<(String, String, String)>,
headers: HeaderMap,
jar: CookieJar,
uri: Uri,
) -> impl IntoResponse {
- let room_id = format!("{room_short}/{room_slug}");
+ let Some(room_id) = slug_types::room_id_from_route_segment(&room_key) else {
+ return (StatusCode::NOT_FOUND, "bad room path").into_response();
+ };
let reduced = state.reduced.read().await;
let user = optional_principal(&headers, &jar, &reduced);
if !user_can_view_room(&reduced, &room_id, user.as_deref()) {
diff --git a/server/src/html/forum/views.rs b/server/src/html/forum/views.rs
index be5df1745ef580891a167c23c3dd6c06f804f295..1ec421f84335cbfe7f9db775b73a8ed24b197174 100644
--- a/server/src/html/forum/views.rs
+++ b/server/src/html/forum/views.rs
@@ -183,16 +183,18 @@ pub async fn thread_view(
thread_view_inner(state, tag, q, ThreadNav::public(), headers, jar, uri).await
}
-/// Room thread — `/r/:short/:slug/t/:tag`
+/// Room thread — `/r/:room_key/t/:tag` (`room_key` = `{short}{slug}`).
pub async fn room_thread_view(
State(state): State<AppState>,
- Path((room_short, room_slug, tag)): Path<(String, String, String)>,
+ Path((room_key, tag)): Path<(String, String)>,
Query(q): Query<ThreadViewQuery>,
headers: HeaderMap,
jar: CookieJar,
uri: Uri,
) -> impl IntoResponse {
- let room_id = format!("{room_short}/{room_slug}");
+ let Some(room_id) = slug_types::room_id_from_route_segment(&room_key) else {
+ return (StatusCode::NOT_FOUND, "bad room path").into_response();
+ };
let reduced = state.reduced.read().await;
let user = optional_principal(&headers, &jar, &reduced);
if !user_can_view_room(&reduced, &room_id, user.as_deref()) {
@@ -226,15 +228,17 @@ pub(super) fn room_not_found_page(jar: &CookieJar, uri: &Uri) -> impl IntoRespon
(StatusCode::NOT_FOUND, Html(page.into_string()))
}
-/// Private room index — `/r/:short/:slug`
+/// Private room index — `/r/:room_key`
pub async fn room_page(
State(state): State<AppState>,
- Path((room_short, room_slug)): Path<(String, String)>,
+ Path(room_key): Path<String>,
headers: HeaderMap,
jar: CookieJar,
uri: Uri,
) -> impl IntoResponse {
- let room_id = format!("{room_short}/{room_slug}");
+ let Some(room_id) = slug_types::room_id_from_route_segment(&room_key) else {
+ return (StatusCode::NOT_FOUND, "room not found").into_response();
+ };
let now = now_ms();
let reduced = state.reduced.read().await;
if !reduced.rooms.contains(&room_id) {
@@ -266,7 +270,10 @@ pub async fn room_page(
let audit_cli = format!("npx slugsocial private {room_id} audit");
drop(reduced);
- let slug_display = room_slug.as_str();
+ let slug_display = room_id
+ .split_once('/')
+ .map(|(_, slug)| slug)
+ .unwrap_or(room_id.as_str());
let page = layout(
&format!("room {slug_display} — slug.social"),
"view-thread",
diff --git a/server/src/html/garden.rs b/server/src/html/garden.rs
index 423f23fd8c9ad7b7f454d6ea7a9a7607a4c9c5b9..e615dd356bcf634232d85610c0a26235ead125fd 100644
--- a/server/src/html/garden.rs
+++ b/server/src/html/garden.rs
@@ -309,12 +309,14 @@ pub async fn external_ontology_path(
pub async fn room_garden_index(
State(state): State<AppState>,
- Path((room_short, room_slug)): Path<(String, String)>,
+ Path(room_key): Path<String>,
headers: HeaderMap,
jar: CookieJar,
uri: Uri,
) -> impl IntoResponse {
- let room_id = format!("{room_short}/{room_slug}");
+ let Some(room_id) = slug_types::room_id_from_route_segment(&room_key) else {
+ return (StatusCode::NOT_FOUND, "bad room path").into_response();
+ };
let Some(nav) = ThreadNav::from_room_id(&room_id) else {
return (StatusCode::NOT_FOUND, "bad room path").into_response();
};
@@ -341,12 +343,14 @@ pub async fn room_garden_index(
pub async fn room_external_garden_index(
State(state): State<AppState>,
- Path((room_short, room_slug)): Path<(String, String)>,
+ Path(room_key): Path<String>,
headers: HeaderMap,
jar: CookieJar,
uri: Uri,
) -> impl IntoResponse {
- let room_id = format!("{room_short}/{room_slug}");
+ let Some(room_id) = slug_types::room_id_from_route_segment(&room_key) else {
+ return (StatusCode::NOT_FOUND, "bad room path").into_response();
+ };
let Some(nav) = ThreadNav::from_room_id(&room_id) else {
return (StatusCode::NOT_FOUND, "bad room path").into_response();
};
@@ -416,12 +420,14 @@ pub async fn room_external_garden_index(
pub async fn room_external_ontology_path(
State(state): State<AppState>,
- Path((room_short, room_slug, path)): Path<(String, String, String)>,
+ Path((room_key, path)): Path<(String, String)>,
headers: HeaderMap,
jar: CookieJar,
uri: Uri,
) -> impl IntoResponse {
- let room_id = format!("{room_short}/{room_slug}");
+ let Some(room_id) = slug_types::room_id_from_route_segment(&room_key) else {
+ return (StatusCode::NOT_FOUND, "bad room path").into_response();
+ };
let Some(nav) = ThreadNav::from_room_id(&room_id) else {
return (StatusCode::NOT_FOUND, "bad room path").into_response();
};
@@ -442,12 +448,14 @@ pub async fn room_external_ontology_path(
pub async fn room_ontology_path(
State(state): State<AppState>,
- Path((room_short, room_slug, path)): Path<(String, String, String)>,
+ Path((room_key, path)): Path<(String, String)>,
headers: HeaderMap,
jar: CookieJar,
uri: Uri,
) -> impl IntoResponse {
- let room_id = format!("{room_short}/{room_slug}");
+ let Some(room_id) = slug_types::room_id_from_route_segment(&room_key) else {
+ return (StatusCode::NOT_FOUND, "bad room path").into_response();
+ };
let Some(nav) = ThreadNav::from_room_id(&room_id) else {
return (StatusCode::NOT_FOUND, "bad room path").into_response();
};
diff --git a/server/src/html/search.rs b/server/src/html/search.rs
index 43e6ebf36cf0fe72c96f0f9d850bea51ac094c43..f01732f7edb32c68fcc10c39545c8f56476adb27 100644
--- a/server/src/html/search.rs
+++ b/server/src/html/search.rs
@@ -351,8 +351,8 @@ fn render_search_results(results: &SearchResults, query: &str) -> Markup {
ul class="search-posts" {
@for r in &results.posts {
@let (post_href, post_label) = if let Some((room, tag)) = r.thread.split_once("/#") {
- if let Some((short, slug)) = room.split_once('/') {
- (format!("/r/{short}/{slug}/t/{tag}"), format!("{room}/#{tag}"))
+ if let Some(seg) = slug_types::room_route_segment(room) {
+
… preview truncated; 35,812 characters omittedB — c_c124c217f89c (tommy-mor)
message
[8d8230d1] reddit
diff preview
diff --git a/.gitignore b/.gitignore
index 4c7073f9fac0c30fd2050d79a60ef447af58ebeb..ada462e900d24a3a6d08165d158c80f79c35a5a5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,3 +7,4 @@
data/
repomix-output.xml
dev-data/
+.env
diff --git a/server/Cargo.toml b/server/Cargo.toml
index 7906a8547d56b8e6a48ef59c37aa82a8510fdee9..4677fedcb45292eebebe7e9cf6ce2f5738f18ddf 100644
--- a/server/Cargo.toml
+++ b/server/Cargo.toml
@@ -16,6 +16,7 @@ tower = "0.5"
tower-http = { version = "0.5", features = ["trace"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
+reqwest = { version = "0.12", features = ["json"] }
[dev-dependencies]
reqwest = { version = "0.12", features = ["json"] }
diff --git a/server/src/html/mod.rs b/server/src/html/mod.rs
index 2864407ed6e8ec284a1dc663acf1805534566b62..df6505021d9f446c2b453e20e3eb3cf696a111f9 100644
--- a/server/src/html/mod.rs
+++ b/server/src/html/mod.rs
@@ -272,5 +272,16 @@ pub async fn home(State(state): State<AppState>, uri: Uri) -> impl IntoResponse
pub async fn browse(State(state): State<AppState>, uri: Uri) -> impl IntoResponse {
let item = ItemId::from_browse_uri(uri.path()).unwrap_or(ItemId::root());
+ if item.as_str().starts_with("reddit.com") {
+ let needs_fetch = {
+ let tree = state.tree.read().await;
+ tree.get(&item)
+ .map(|n| n.data.is_none())
+ .unwrap_or(true)
+ };
+ if needs_fetch {
+ state.reddit.request_fetch(item.clone());
+ }
+ }
item_page(state, uri, item).await
}
diff --git a/server/src/reddit.rs b/server/src/reddit.rs
index d203dca09245daf869b3aa942898447700ae69fb..90053ad03b1d7c8e94f325dd4ee64c2b4f7da900 100644
--- a/server/src/reddit.rs
+++ b/server/src/reddit.rs
@@ -1,4 +1,12 @@
-//! Reddit API import (async, decoupled from UI request path).
+//! Reddit API import via a single background worker (rate limits, dedup, backoff).
+
+use std::collections::{HashMap, HashSet};
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+
+use reqwest::{header, Client, StatusCode};
+use serde::Deserialize;
+use tokio::sync::{mpsc, RwLock};
use crate::{
path_types::ItemId,
@@ -10,12 +18,401 @@ pub fn ensure_partial_tree(tree: &mut GlobalTree, id: &ItemId) {
tree.ensure_path(id);
}
-/// Placeholder for Reddit JSON import. Returns entity data when implemented.
-pub async fn fetch_reddit_entity(_id: &ItemId) -> Option<EntityData> {
- None
+pub struct RedditCommand {
+ pub id: ItemId,
+}
+
+#[derive(Clone)]
+pub struct RedditBroker {
+ tx: mpsc::Sender<RedditCommand>,
+}
+
+#[derive(Clone)]
+struct RedditCredentials {
+ client_id: String,
+ client_secret: String,
+}
+
+struct OAuthToken {
+ access_token: String,
+ expires_at: Instant,
+}
+
+impl RedditBroker {
+ pub fn spawn(tree: Arc<RwLock<GlobalTree>>, user_agent: &str) -> Self {
+ let (tx, rx) = mpsc::channel(100);
+
+ let mut headers = header::HeaderMap::new();
+ headers.insert(
+ header::USER_AGENT,
+ header::HeaderValue::from_str(user_agent).expect("valid user agent"),
+ );
+
+ let client = Client::builder()
+ .default_headers(headers)
+ .timeout(Duration::from_secs(15))
+ .build()
+ .expect("reqwest client");
+
+ let creds = RedditCredentials::from_env();
+ tokio::spawn(reddit_worker(rx, tree, client, creds));
+
+ Self { tx }
+ }
+
+ /// Fire-and-forget: queue a fetch; worker updates the tree when done.
+ pub fn request_fetch(&self, id: ItemId) {
+ let _ = self.tx.try_send(RedditCommand { id });
+ }
+}
+
+impl RedditCredentials {
+ fn from_env() -> Option<Self> {
+ let client_id = std::env::var("REDDIT_CLIENT_ID").ok()?;
+ let client_secret = std::env::var("REDDIT_CLIENT_SECRET").ok()?;
+ if client_id.is_empty() || client_secret.is_empty() {
+ return None;
+ }
+ Some(Self {
+ client_id,
+ client_secret,
+ })
+ }
+}
+
+pub fn default_user_agent() -> String {
+ std::env::var("REDDIT_USER_AGENT").unwrap_or_else(|_| {
+ "web:sorter2.social:v0.0.1 (by /u/sorter2)".to_string()
+ })
}
-/// Apply fetched entity data to a node (called from async worker).
-pub fn apply_entity(tree: &mut GlobalTree, id: &ItemId, data: EntityData) {
- tree.set_entity_data(id, data);
+async fn reddit_worker(
+ mut rx: mpsc::Receiver<RedditCommand>,
+ tree: Arc<RwLock<GlobalTree>>,
+ client: Client,
+ creds: Option<RedditCredentials>,
+) {
+ let mut in_flight = HashSet::new();
+ let mut recently_fetched: HashMap<ItemId, Instant> = HashMap::new();
+ let mut current_delay = Duration::from_secs(1);
+ let mut oauth: Option<OAuthToken> = None;
+ let cache_ttl = Duration::from_secs(300);
+
+ while let Some(cmd) = rx.recv().await {
+ let now = Instant::now();
+ recently_fetched.retain(|_, t| now.duration_since(*t) < cache_ttl);
+
+ if in_flight.contains(&cmd.id) || recently_fetched.contains_key(&cmd.id) {
+ continue;
+ }
+
+ in_flight.insert(cmd.id.clone());
+ let fetch_id = cmd.id.clone();
+
+ tokio::time::sleep(current_delay).await;
+
+ if let Some(c) = &creds {
+ oauth = ensure_oauth_token(&client, c, oauth.take()).await;
+ }
+
+ let token = oauth.as_ref().map(|t| t.access_token.as_str());
+ let use_oauth = token.is_some();
+
+ match do_fetch(&client, &fetch_id, use_oauth, token).await {
+ Ok(FetchOutcome::Entity(data)) => {
+ let mut w = tree.write().await;
+ w.set_entity_data(&fetch_id, data);
+ recently_fetched.insert(fetch_id.clone(), Instant::now());
+ current_delay = Duration::from_millis(600);
+ }
+ Ok(FetchOutcome::NotFound) => {
+ recently_fetched.insert(fetch_id.clone(), Instant::now());
+ }
+ Ok(FetchOutcome::RateLimited { reset_secs }) => {
+ let wait = Duration::from_secs(reset_secs.max(1));
+ tracing::warn!(
+ "Reddit rate limit for {}; sleeping {}s",
+ fetch_id,
+ wait.as_secs()
+ );
+ tokio::time::sleep(wait).await;
+ current_delay = (current_delay * 2).min(Duration::from_secs(60));
+ }
+ Err(e) => {
+ tracing::warn!("Reddit fetch failed for {}: {}", fetch_id, e);
+ current_delay = (current_delay * 2).min(Duration::from_secs(60));
+ }
+ }
+
+ in_flight.remove(&fetch_id);
+ }
+}
+
+enum FetchOutcome {
+ Entity(EntityData),
+ NotFound,
+ RateLimited { reset_secs: u64 },
+}
+
+async fn ensure_oauth_token(
+ client: &Client,
+ creds: &RedditCredentials,
+ existing: Option<OAuthToken>,
+) -> Option<OAuthToken> {
+ if let Some(t) = existing {
+ if Instant::now() < t.expires_at - Duration::from_secs(60) {
+ return Some(t);
+ }
+ }
+
+ let resp = client
+ .post("https://www.reddit.com/api/v1/access_token")
+ .basic_auth(&creds.client_id, Some(&creds.client_secret))
+ .form(&[("grant_type", "client_credentials")])
+ .send()
+ .await;
+
+ let resp = match resp {
+ Ok(r) => r,
+ Err(e) => {
+ tracing::warn!("Reddit OAuth token request failed: {e}");
+ return None;
+ }
+ };
+
+ if !resp.status().is_success() {
+ tracing::warn!("Reddit OAuth token HTTP {}", resp.status());
+ return None;
+ }
+
+ #[derive(Deserialize)]
+ struct TokenResponse {
+ access_token: String,
+ expires_in: u64,
+ }
+
+ let body: TokenResponse = match resp.json().await {
+ Ok(b) => b,
+ Err(e) => {
+ tracing::warn!("Reddit OAuth token parse failed: {e}");
+ return None;
+ }
+ };
+
+ Some(OAuthToken {
+ access_token: body.access_token,
+ expires_at: Instant::now() + Duration::from_secs(body.expires_in),
+ })
+}
+
+async fn do_fetch(
+ client: &Client,
+ id: &ItemId,
+ use_oauth: bool,
+ bearer: Option<&str>,
+) -> Result<FetchOutcome, String> {
+ let url = map_item_to_reddit_api(id, use_oauth);
+ if url.is_empty() {
+ return Ok(FetchOutcome::NotFound);
+ }
+
+ let mut req = client.get(&url);
+ if let Some(token) = bearer {
+ req = req.bearer_auth(token);
+ }
+
+ let resp = req.send().await.map_err(|e| e.to_string())?;
+
+ if resp.status() == StatusCode::TOO_MANY_REQUESTS {
+ let reset = rate_limit_reset_secs(&resp);
+ return Ok(FetchOutcome::RateLimited { reset_secs: reset });
+ }
+
+ if resp.status() == StatusCode::SERVICE_UNAVAILABLE {
+ return Err("Reddit unavailable (503)".to_string());
+ }
+
+ if !resp.status().is_success() {
+ return Ok(FetchOutcome::NotFound);
+ }
+
+ if rate_limit_remaining(&resp) == Some(0) {
+ let reset = rate_limit_reset_secs(&resp);
+ return Ok(FetchOutcome::RateLimited { reset_secs: reset });
+ }
+
+ let bytes = resp.bytes().await.map_err(|e| e.to_string())?;
+ Ok(parse_reddit_json(id, &bytes)
+ .map(FetchOutcome::Entity)
+ .unwrap_or(FetchOutcome::NotFound))
+}
+
+fn rate_limit_remaining(resp: &reqwest::Response) -> Option<u64> {
+ resp.headers()
+ .get("x-ratelimit-remaining")
+ .and_then(|v| v.to_str().ok())
+ .and_then(|s| s.parse::<f64>().ok())
+ .map(|f| f.floor() as u64)
+}
+
+fn rate_limit_reset_secs(resp: &reqwest::Response) -> u64 {
+ resp.headers()
+ .get("x-ratelimit-reset")
+ .and_then(|v| v.to_str().ok())
+ .and_then(|s| s.parse::<f64>().ok())
+ .map(|f| f.ceil() as u64)
+ .unwrap_or(5)
+}
+
+/// Map canonical item id to Reddit JSON API URL.
+pub fn map_item_to_reddit_api(id: &ItemId, oauth: bool) -> String {
+ let path = id.as_str();
+ if !path.starts_with("reddit.com/") && path != "reddit.com" {
+ return String::new();
+ }
+
+ let base = if oauth {
+ "https://oauth.reddit.com"
+ } else {
+ "https://www.reddit.com"
+ };
+
+ let segments: Vec<&str> = path.split('/').collect();
+
+ if let Some(i) = segments.iter().position(|&p| p == "comments") {
+ if segments.len() > i + 1 {
+ let api_path = segments[1..=i + 1].join("/");
+ return format!("{base}/{api_path}.json?raw_json=1");
+ }
+ }
+
+ if segments.len() == 3 && segments[1] == "r" {
+ return format!("{base}/r/{}/about.json?raw_json=1", segments[2]);
+ }
+
+ String::new()
+}
+
+fn parse_reddit_json(id: &ItemId, bytes: &[u8]) -> Option<EntityData> {
+ let v: serde_json::Value = serde_json::from_slice(bytes).ok()?;
+ let segments: Vec<&str> = id.as_str().split('/').collect();
+
+ if segments.iter().any(|&p| p == "comments") {
+ parse_post_listing(&v)
+ } else {
+ parse_subreddit_about(&v)
+ }
+}
+
+fn parse_subreddit_about(v: &serde_json::Value) -> Option<EntityData> {
+ let data = v.get("data")?;
+ let title = data
+ .get("title")
+ .or_else(|| data.get("display_name"))
+ .and_then(|t| t.as_str())?
+ .to_string();
+ let body_html = data
+ .get("public_description_html")
+ .or_else(|| data.get("public_description"))
+ .and_then(|t| t.as_str())
+ .map(|s| s.to_string());
+ let thumb_url = data
+ .get("icon_img")
+ .or_else(|| data.get("community_icon"))
+ .and_then(|t| t.as_str())
+ .filter(|s| !s.is_empty())
+ .map(|s| s.to_string());
+
+ Some(EntityData {
+ title,
+ author: None,
+ body_html,
+ thumb_url,
+ })
+}
+
+fn parse_post_listing(v:
… preview truncated; 4,570 characters omittedHardlinks — judgments / attempts / prompt
judgments
attempts
Prompt text is loaded only by the download route.