Side A fixes a real design flaw in B's auto-fetch-on-browse behavior by making imports explicit user actions, persists the full API payload through the event log for correct replay/audit (via a new EntityImported event and apply_entity_raw), and adds solid regression tests (replay test, fixture-based Reddit test, and an end-to-end mock-API integration test). Side B is the initial (and reasonable) implementation of the Reddit worker/broker, but it auto-fetches on every browse without persisting raw payloads, which A explicitly corrects and extends with more durable, testable design.
constitution · epochs · watch · epoch 3
comparison
c_e57094c6229a (tommy-mor) vs c_c124c217f89c (tommy-mor)
download prompt · raw event · cmp_b74307665eb037
council reasoning
A turns B’s in-memory-only auto-fetch into a durable design: EntityImported events store full API payloads, replay restores entity_raw + derived views, and fetch is an explicit UI action with configurable API bases. B still matters as the first real RedditBroker (OAuth, rate limits, URL mapping, parsers), but much of that surface is reshaped by A, and A adds the fixture plus end-to-end mock import test that locks the behavior in.
Side A substantially evolves the Reddit import design by replacing automatic fetch-on-browse with an explicit user-triggered fetch UI, persisting full API payloads as `EntityImported` events, replaying them into state, adding configurable API endpoints, dotenv support, and tests for replay and end-to-end import. Side B mainly introduces the initial background Reddit fetch worker and automatic fetching on browse, but lacks the event-sourced persistence, explicit fetch workflow, configurability, and broader architectural improvements that make Side A's changes more durable.
sides
A — c_e57094c6229a (tommy-mor)
message
[40b975bf] nice
diff preview
diff --git a/Cargo.lock b/Cargo.lock
index 266e876bb7ccbe788beb1d5bd53ad5b45ee5825b..2cea973082716e761ef6f5dd5886acc08ff9aac0 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -222,6 +222,12 @@ dependencies = [
"syn",
]
+[[package]]
+name = "dotenvy"
+version = "0.15.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1aaf95b3e5c8f23aa320147307562d361db0ae0d51242340f558153b4eb2439b"
+
[[package]]
name = "encoding_rs"
version = "0.8.35"
@@ -1238,6 +1244,7 @@ version = "0.0.1"
dependencies = [
"axum",
"axum-extra",
+ "dotenvy",
"maud",
"reqwest",
"serde",
diff --git a/server/Cargo.toml b/server/Cargo.toml
index 4677fedcb45292eebebe7e9cf6ce2f5738f18ddf..bd600138b613bd0f546bdec217a5334cdcb20aa5 100644
--- a/server/Cargo.toml
+++ b/server/Cargo.toml
@@ -17,6 +17,7 @@ tower-http = { version = "0.5", features = ["trace"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
reqwest = { version = "0.12", features = ["json"] }
+dotenvy = "0.15"
[dev-dependencies]
reqwest = { version = "0.12", features = ["json"] }
diff --git a/server/src/api/ui_html.rs b/server/src/api/ui_html.rs
index d2024bd4582bcc8482b461b2ba4fedbd8bff7c66..b33a84e8bb5e817b26592868d88090e6d664d950 100644
--- a/server/src/api/ui_html.rs
+++ b/server/src/api/ui_html.rs
@@ -6,7 +6,7 @@ use axum::{
use std::collections::HashMap;
use crate::{
- html::{input_panel, js_string_literal, ranking_panel, JsBuilder},
+ html::{entity_section, input_panel, js_string_literal, ranking_panel, JsBuilder},
parser::parse_reddit_url,
path_types::ItemId,
reddit::ensure_partial_tree,
@@ -87,6 +87,20 @@ pub async fn post_ui_html(
.into_response()
}
},
+ HtmlUiAction::FetchEntity { item } => {
+ let id = parse_item_param(&item);
+ if id.is_root() {
+ return ui_js_warn("nothing to fetch for the root").into_response();
+ }
+ state.queue_entity_fetch(id.clone());
+ let tree = state.tree.read().await;
+ let empty = crate::reducer::NodeState::default();
+ let node = tree.get(&id).unwrap_or(&empty);
+ let panel = entity_section(&id, node, true);
+ JsBuilder::new()
+ .morph_selector("#entity-section", panel)
+ .into_response()
+ },
}
}
diff --git a/server/src/events.rs b/server/src/events.rs
index ed5be6b13b9d46e838831d6ce0f96f569b401730..07ce24b5e56cf72b0b442c3c3241efbf6c3b006a 100644
--- a/server/src/events.rs
+++ b/server/src/events.rs
@@ -1,4 +1,5 @@
use serde::{Deserialize, Serialize};
+use serde_json::Value;
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type", rename_all = "snake_case")]
@@ -18,4 +19,10 @@ pub enum Event {
},
/// Register a node path in the fractal tree (no external fetch).
NodeEnsured { id: String },
+ /// Full upstream API payload for a node (domain-specific view derived at replay/render time).
+ EntityImported {
+ id: String,
+ ts: i64,
+ payload: Value,
+ },
}
diff --git a/server/src/html/mod.rs b/server/src/html/mod.rs
index df6505021d9f446c2b453e20e3eb3cf696a111f9..caf1309c8d93b47104499c57f9cc35ee7631fbb9 100644
--- a/server/src/html/mod.rs
+++ b/server/src/html/mod.rs
@@ -10,6 +10,7 @@ use crate::{
form_template::template_json_compact,
path_types::ItemId,
ranking::{top_bottom, RankedItem},
+ reddit::is_fetchable,
reducer::{GroupState, NodeState},
state::AppState,
ui_action::UI_RPC_FIELD,
@@ -151,7 +152,7 @@ pub fn breadcrumb_path(item: &ItemId) -> Markup {
fn entity_panel(node: &NodeState) -> Markup {
html! {
@if let Some(data) = &node.data {
- section id="entity-panel" class="demo-panel entity-card" {
+ div id="entity-panel" class="entity-card" {
h2 { (data.title) }
@if let Some(author) = &data.author {
p class="muted small" { "by " (author) }
@@ -164,6 +165,42 @@ fn entity_panel(node: &NodeState) -> Markup {
}
}
+/// Reddit/API import control — only shown on fetchable pages; never auto-fires.
+pub fn fetch_entity_panel(item: &ItemId, has_data: bool, fetching: bool) -> Markup {
+ if !is_fetchable(item) {
+ return html! {};
+ }
+ let label = if fetching {
+ "Fetching…"
+ } else if has_data {
+ "Fetch more"
+ } else {
+ "Fetch from Reddit"
+ };
+ let rpc = template_json_compact(&serde_json::json!({
+ "action": "fetch_entity",
+ "item": item.as_str(),
+ }))
+ .expect("fetch_entity rpc template");
+ html! {
+ form method="post" action="/ui" id="fetch-entity-form" class="fetch-entity-form" {
+ input type="hidden" name=(UI_RPC_FIELD) value=(rpc);
+ button type="submit" class="btn-secondary" disabled=(fetching) { (label) }
+ }
+ }
+}
+
+/// Entity card + explicit fetch control (morphed as `#entity-section`).
+pub fn entity_section(item: &ItemId, node: &NodeState, fetching: bool) -> Markup {
+ let has_data = node.data.is_some();
+ html! {
+ section id="entity-section" class="demo-panel" {
+ (entity_panel(node))
+ (fetch_entity_panel(item, has_data, fetching))
+ }
+ }
+}
+
fn rank_list(label: &str, items: &[RankedItem], start_rank: usize) -> Markup {
html! {
@if !items.is_empty() {
@@ -260,7 +297,7 @@ async fn item_page(state: AppState, uri: Uri, item: ItemId) -> Markup {
h1 { "sorter" }
(input_panel("", None))
(breadcrumb_path(&item))
- (entity_panel(node))
+ (entity_section(&item, node, false))
(ranking_panel(&item, group))
};
layout("sorter2", body, views)
@@ -272,16 +309,5 @@ pub async fn home(State(state): State<AppState>, uri: Uri) -> impl IntoResponse
pub async fn browse(State(state): State<AppState>, uri: Uri) -> impl IntoResponse {
let item = ItemId::from_browse_uri(uri.path()).unwrap_or(ItemId::root());
- if item.as_str().starts_with("reddit.com") {
- let needs_fetch = {
- let tree = state.tree.read().await;
- tree.get(&item)
- .map(|n| n.data.is_none())
- .unwrap_or(true)
- };
- if needs_fetch {
- state.reddit.request_fetch(item.clone());
- }
- }
item_page(state, uri, item).await
}
diff --git a/server/src/main.rs b/server/src/main.rs
index c22ec6c9f5358e5ec99fb83210dc351938505a93..1f0cddc39302b35b0cd6a6219f44c9d59202facf 100644
--- a/server/src/main.rs
+++ b/server/src/main.rs
@@ -2,6 +2,10 @@ use sorter2_server::state::AppConfig;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+ if std::env::var("SORTER2_SKIP_DOTENV").is_err() {
+ let _ = dotenvy::dotenv();
+ }
+
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env()
diff --git a/server/src/reddit.rs b/server/src/reddit.rs
index 90053ad03b1d7c8e94f325dd4ee64c2b4f7da900..ff0f01e57b18af878eb5be3efc47204a7673589d 100644
--- a/server/src/reddit.rs
+++ b/server/src/reddit.rs
@@ -6,11 +6,15 @@ use std::time::{Duration, Instant};
use reqwest::{header, Client, StatusCode};
use serde::Deserialize;
+use serde_json::Value;
use tokio::sync::{mpsc, RwLock};
use crate::{
+ event_log::EventLog,
+ events::Event,
+ html::now_ms,
path_types::ItemId,
- reducer::{EntityData, GlobalTree},
+ reducer::GlobalTree,
};
/// Bootstrap blank nodes along a URL path so breadcrumbs and voting work before fetch.
@@ -20,6 +24,8 @@ pub fn ensure_partial_tree(tree: &mut GlobalTree, id: &ItemId) {
pub struct RedditCommand {
pub id: ItemId,
+ /// User-initiated fetch bypasses the in-memory "recently fetched" cache.
+ pub force: bool,
}
#[derive(Clone)]
@@ -33,19 +39,31 @@ struct RedditCredentials {
client_secret: String,
}
+#[derive(Clone)]
+pub struct RedditApiConfig {
+ pub api_base: String,
+ pub oauth_base: String,
+ pub user_agent: String,
+ creds: Option<RedditCredentials>,
+}
+
struct OAuthToken {
access_token: String,
expires_at: Instant,
}
impl RedditBroker {
- pub fn spawn(tree: Arc<RwLock<GlobalTree>>, user_agent: &str) -> Self {
+ pub fn spawn(
+ tree: Arc<RwLock<GlobalTree>>,
+ event_log: Arc<EventLog>,
+ config: RedditApiConfig,
+ ) -> Self {
let (tx, rx) = mpsc::channel(100);
let mut headers = header::HeaderMap::new();
headers.insert(
header::USER_AGENT,
- header::HeaderValue::from_str(user_agent).expect("valid user agent"),
+ header::HeaderValue::from_str(&config.user_agent).expect("valid user agent"),
);
let client = Client::builder()
@@ -54,22 +72,38 @@ impl RedditBroker {
.build()
.expect("reqwest client");
- let creds = RedditCredentials::from_env();
- tokio::spawn(reddit_worker(rx, tree, client, creds));
+ tokio::spawn(reddit_worker(rx, tree, event_log, client, config));
Self { tx }
}
- /// Fire-and-forget: queue a fetch; worker updates the tree when done.
- pub fn request_fetch(&self, id: ItemId) {
- let _ = self.tx.try_send(RedditCommand { id });
+ /// Queue a fetch; drops when the channel is full (backpressure).
+ pub fn request_fetch(&self, id: ItemId, force: bool) {
+ let _ = self.tx.try_send(RedditCommand { id, force });
+ }
+}
+
+impl RedditApiConfig {
+ pub fn from_env() -> Self {
+ Self {
+ api_base: reddit_api_base(),
+ oauth_base: reddit_oauth_base(),
+ user_agent: default_user_agent(),
+ creds: RedditCredentials::from_env(),
+ }
}
}
impl RedditCredentials {
+ /// Reddit's OAuth docs call these "client id" and "client secret"; the app
+ /// registration UI often labels them "app id" / "app secret" — same values.
fn from_env() -> Option<Self> {
- let client_id = std::env::var("REDDIT_CLIENT_ID").ok()?;
- let client_secret = std::env::var("REDDIT_CLIENT_SECRET").ok()?;
+ let client_id = std::env::var("REDDIT_CLIENT_ID")
+ .or_else(|_| std::env::var("REDDIT_APP_ID"))
+ .ok()?;
+ let client_secret = std::env::var("REDDIT_CLIENT_SECRET")
+ .or_else(|_| std::env::var("REDDIT_APP_SECRET"))
+ .ok()?;
if client_id.is_empty() || client_secret.is_empty() {
return None;
}
@@ -80,29 +114,63 @@ impl RedditCredentials {
}
}
+pub fn reddit_api_base() -> String {
+ std::env::var("REDDIT_API_BASE").unwrap_or_else(|_| "https://www.reddit.com".into())
+}
+
+pub fn reddit_oauth_base() -> String {
+ std::env::var("REDDIT_OAUTH_BASE").unwrap_or_else(|_| "https://www.reddit.com".into())
+}
+
pub fn default_user_agent() -> String {
std::env::var("REDDIT_USER_AGENT").unwrap_or_else(|_| {
"web:sorter2.social:v0.0.1 (by /u/sorter2)".to_string()
})
}
+/// True when this node can be loaded from the Reddit JSON API.
+pub fn is_fetchable(id: &ItemId) -> bool {
+ !map_item_to_reddit_api(id, "https://example.com").is_empty()
+}
+
+/// Derive UI-facing fields from a stored payload (Reddit-specific when under reddit.com).
+pub fn entity_view_from_payload(id: &ItemId, payload: &Value) -> Option<crate::reducer::EntityData> {
+ if id.as_str().starts_with("reddit.com") {
+ return parse_reddit_view(id, payload);
+ }
+ None
+}
+
+/// Apply a full API payload to the in-memory tree (view derived for known domains).
+pub fn apply_entity_import(tree: &mut GlobalTree, id: &ItemId, payload: Value) {
+ let view = entity_view_from_payload(id, &payload);
+ tree.apply_entity_raw(id, payload, view);
+}
+
async fn red
… preview truncated; 22,673 characters omittedB — c_c124c217f89c (tommy-mor)
message
[8d8230d1] reddit
diff preview
diff --git a/.gitignore b/.gitignore
index 4c7073f9fac0c30fd2050d79a60ef447af58ebeb..ada462e900d24a3a6d08165d158c80f79c35a5a5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,3 +7,4 @@
data/
repomix-output.xml
dev-data/
+.env
diff --git a/server/Cargo.toml b/server/Cargo.toml
index 7906a8547d56b8e6a48ef59c37aa82a8510fdee9..4677fedcb45292eebebe7e9cf6ce2f5738f18ddf 100644
--- a/server/Cargo.toml
+++ b/server/Cargo.toml
@@ -16,6 +16,7 @@ tower = "0.5"
tower-http = { version = "0.5", features = ["trace"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
+reqwest = { version = "0.12", features = ["json"] }
[dev-dependencies]
reqwest = { version = "0.12", features = ["json"] }
diff --git a/server/src/html/mod.rs b/server/src/html/mod.rs
index 2864407ed6e8ec284a1dc663acf1805534566b62..df6505021d9f446c2b453e20e3eb3cf696a111f9 100644
--- a/server/src/html/mod.rs
+++ b/server/src/html/mod.rs
@@ -272,5 +272,16 @@ pub async fn home(State(state): State<AppState>, uri: Uri) -> impl IntoResponse
pub async fn browse(State(state): State<AppState>, uri: Uri) -> impl IntoResponse {
let item = ItemId::from_browse_uri(uri.path()).unwrap_or(ItemId::root());
+ if item.as_str().starts_with("reddit.com") {
+ let needs_fetch = {
+ let tree = state.tree.read().await;
+ tree.get(&item)
+ .map(|n| n.data.is_none())
+ .unwrap_or(true)
+ };
+ if needs_fetch {
+ state.reddit.request_fetch(item.clone());
+ }
+ }
item_page(state, uri, item).await
}
diff --git a/server/src/reddit.rs b/server/src/reddit.rs
index d203dca09245daf869b3aa942898447700ae69fb..90053ad03b1d7c8e94f325dd4ee64c2b4f7da900 100644
--- a/server/src/reddit.rs
+++ b/server/src/reddit.rs
@@ -1,4 +1,12 @@
-//! Reddit API import (async, decoupled from UI request path).
+//! Reddit API import via a single background worker (rate limits, dedup, backoff).
+
+use std::collections::{HashMap, HashSet};
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+
+use reqwest::{header, Client, StatusCode};
+use serde::Deserialize;
+use tokio::sync::{mpsc, RwLock};
use crate::{
path_types::ItemId,
@@ -10,12 +18,401 @@ pub fn ensure_partial_tree(tree: &mut GlobalTree, id: &ItemId) {
tree.ensure_path(id);
}
-/// Placeholder for Reddit JSON import. Returns entity data when implemented.
-pub async fn fetch_reddit_entity(_id: &ItemId) -> Option<EntityData> {
- None
+pub struct RedditCommand {
+ pub id: ItemId,
+}
+
+#[derive(Clone)]
+pub struct RedditBroker {
+ tx: mpsc::Sender<RedditCommand>,
+}
+
+#[derive(Clone)]
+struct RedditCredentials {
+ client_id: String,
+ client_secret: String,
+}
+
+struct OAuthToken {
+ access_token: String,
+ expires_at: Instant,
+}
+
+impl RedditBroker {
+ pub fn spawn(tree: Arc<RwLock<GlobalTree>>, user_agent: &str) -> Self {
+ let (tx, rx) = mpsc::channel(100);
+
+ let mut headers = header::HeaderMap::new();
+ headers.insert(
+ header::USER_AGENT,
+ header::HeaderValue::from_str(user_agent).expect("valid user agent"),
+ );
+
+ let client = Client::builder()
+ .default_headers(headers)
+ .timeout(Duration::from_secs(15))
+ .build()
+ .expect("reqwest client");
+
+ let creds = RedditCredentials::from_env();
+ tokio::spawn(reddit_worker(rx, tree, client, creds));
+
+ Self { tx }
+ }
+
+ /// Fire-and-forget: queue a fetch; worker updates the tree when done.
+ pub fn request_fetch(&self, id: ItemId) {
+ let _ = self.tx.try_send(RedditCommand { id });
+ }
+}
+
+impl RedditCredentials {
+ fn from_env() -> Option<Self> {
+ let client_id = std::env::var("REDDIT_CLIENT_ID").ok()?;
+ let client_secret = std::env::var("REDDIT_CLIENT_SECRET").ok()?;
+ if client_id.is_empty() || client_secret.is_empty() {
+ return None;
+ }
+ Some(Self {
+ client_id,
+ client_secret,
+ })
+ }
+}
+
+pub fn default_user_agent() -> String {
+ std::env::var("REDDIT_USER_AGENT").unwrap_or_else(|_| {
+ "web:sorter2.social:v0.0.1 (by /u/sorter2)".to_string()
+ })
}
-/// Apply fetched entity data to a node (called from async worker).
-pub fn apply_entity(tree: &mut GlobalTree, id: &ItemId, data: EntityData) {
- tree.set_entity_data(id, data);
+async fn reddit_worker(
+ mut rx: mpsc::Receiver<RedditCommand>,
+ tree: Arc<RwLock<GlobalTree>>,
+ client: Client,
+ creds: Option<RedditCredentials>,
+) {
+ let mut in_flight = HashSet::new();
+ let mut recently_fetched: HashMap<ItemId, Instant> = HashMap::new();
+ let mut current_delay = Duration::from_secs(1);
+ let mut oauth: Option<OAuthToken> = None;
+ let cache_ttl = Duration::from_secs(300);
+
+ while let Some(cmd) = rx.recv().await {
+ let now = Instant::now();
+ recently_fetched.retain(|_, t| now.duration_since(*t) < cache_ttl);
+
+ if in_flight.contains(&cmd.id) || recently_fetched.contains_key(&cmd.id) {
+ continue;
+ }
+
+ in_flight.insert(cmd.id.clone());
+ let fetch_id = cmd.id.clone();
+
+ tokio::time::sleep(current_delay).await;
+
+ if let Some(c) = &creds {
+ oauth = ensure_oauth_token(&client, c, oauth.take()).await;
+ }
+
+ let token = oauth.as_ref().map(|t| t.access_token.as_str());
+ let use_oauth = token.is_some();
+
+ match do_fetch(&client, &fetch_id, use_oauth, token).await {
+ Ok(FetchOutcome::Entity(data)) => {
+ let mut w = tree.write().await;
+ w.set_entity_data(&fetch_id, data);
+ recently_fetched.insert(fetch_id.clone(), Instant::now());
+ current_delay = Duration::from_millis(600);
+ }
+ Ok(FetchOutcome::NotFound) => {
+ recently_fetched.insert(fetch_id.clone(), Instant::now());
+ }
+ Ok(FetchOutcome::RateLimited { reset_secs }) => {
+ let wait = Duration::from_secs(reset_secs.max(1));
+ tracing::warn!(
+ "Reddit rate limit for {}; sleeping {}s",
+ fetch_id,
+ wait.as_secs()
+ );
+ tokio::time::sleep(wait).await;
+ current_delay = (current_delay * 2).min(Duration::from_secs(60));
+ }
+ Err(e) => {
+ tracing::warn!("Reddit fetch failed for {}: {}", fetch_id, e);
+ current_delay = (current_delay * 2).min(Duration::from_secs(60));
+ }
+ }
+
+ in_flight.remove(&fetch_id);
+ }
+}
+
+enum FetchOutcome {
+ Entity(EntityData),
+ NotFound,
+ RateLimited { reset_secs: u64 },
+}
+
+async fn ensure_oauth_token(
+ client: &Client,
+ creds: &RedditCredentials,
+ existing: Option<OAuthToken>,
+) -> Option<OAuthToken> {
+ if let Some(t) = existing {
+ if Instant::now() < t.expires_at - Duration::from_secs(60) {
+ return Some(t);
+ }
+ }
+
+ let resp = client
+ .post("https://www.reddit.com/api/v1/access_token")
+ .basic_auth(&creds.client_id, Some(&creds.client_secret))
+ .form(&[("grant_type", "client_credentials")])
+ .send()
+ .await;
+
+ let resp = match resp {
+ Ok(r) => r,
+ Err(e) => {
+ tracing::warn!("Reddit OAuth token request failed: {e}");
+ return None;
+ }
+ };
+
+ if !resp.status().is_success() {
+ tracing::warn!("Reddit OAuth token HTTP {}", resp.status());
+ return None;
+ }
+
+ #[derive(Deserialize)]
+ struct TokenResponse {
+ access_token: String,
+ expires_in: u64,
+ }
+
+ let body: TokenResponse = match resp.json().await {
+ Ok(b) => b,
+ Err(e) => {
+ tracing::warn!("Reddit OAuth token parse failed: {e}");
+ return None;
+ }
+ };
+
+ Some(OAuthToken {
+ access_token: body.access_token,
+ expires_at: Instant::now() + Duration::from_secs(body.expires_in),
+ })
+}
+
+async fn do_fetch(
+ client: &Client,
+ id: &ItemId,
+ use_oauth: bool,
+ bearer: Option<&str>,
+) -> Result<FetchOutcome, String> {
+ let url = map_item_to_reddit_api(id, use_oauth);
+ if url.is_empty() {
+ return Ok(FetchOutcome::NotFound);
+ }
+
+ let mut req = client.get(&url);
+ if let Some(token) = bearer {
+ req = req.bearer_auth(token);
+ }
+
+ let resp = req.send().await.map_err(|e| e.to_string())?;
+
+ if resp.status() == StatusCode::TOO_MANY_REQUESTS {
+ let reset = rate_limit_reset_secs(&resp);
+ return Ok(FetchOutcome::RateLimited { reset_secs: reset });
+ }
+
+ if resp.status() == StatusCode::SERVICE_UNAVAILABLE {
+ return Err("Reddit unavailable (503)".to_string());
+ }
+
+ if !resp.status().is_success() {
+ return Ok(FetchOutcome::NotFound);
+ }
+
+ if rate_limit_remaining(&resp) == Some(0) {
+ let reset = rate_limit_reset_secs(&resp);
+ return Ok(FetchOutcome::RateLimited { reset_secs: reset });
+ }
+
+ let bytes = resp.bytes().await.map_err(|e| e.to_string())?;
+ Ok(parse_reddit_json(id, &bytes)
+ .map(FetchOutcome::Entity)
+ .unwrap_or(FetchOutcome::NotFound))
+}
+
+fn rate_limit_remaining(resp: &reqwest::Response) -> Option<u64> {
+ resp.headers()
+ .get("x-ratelimit-remaining")
+ .and_then(|v| v.to_str().ok())
+ .and_then(|s| s.parse::<f64>().ok())
+ .map(|f| f.floor() as u64)
+}
+
+fn rate_limit_reset_secs(resp: &reqwest::Response) -> u64 {
+ resp.headers()
+ .get("x-ratelimit-reset")
+ .and_then(|v| v.to_str().ok())
+ .and_then(|s| s.parse::<f64>().ok())
+ .map(|f| f.ceil() as u64)
+ .unwrap_or(5)
+}
+
+/// Map canonical item id to Reddit JSON API URL.
+pub fn map_item_to_reddit_api(id: &ItemId, oauth: bool) -> String {
+ let path = id.as_str();
+ if !path.starts_with("reddit.com/") && path != "reddit.com" {
+ return String::new();
+ }
+
+ let base = if oauth {
+ "https://oauth.reddit.com"
+ } else {
+ "https://www.reddit.com"
+ };
+
+ let segments: Vec<&str> = path.split('/').collect();
+
+ if let Some(i) = segments.iter().position(|&p| p == "comments") {
+ if segments.len() > i + 1 {
+ let api_path = segments[1..=i + 1].join("/");
+ return format!("{base}/{api_path}.json?raw_json=1");
+ }
+ }
+
+ if segments.len() == 3 && segments[1] == "r" {
+ return format!("{base}/r/{}/about.json?raw_json=1", segments[2]);
+ }
+
+ String::new()
+}
+
+fn parse_reddit_json(id: &ItemId, bytes: &[u8]) -> Option<EntityData> {
+ let v: serde_json::Value = serde_json::from_slice(bytes).ok()?;
+ let segments: Vec<&str> = id.as_str().split('/').collect();
+
+ if segments.iter().any(|&p| p == "comments") {
+ parse_post_listing(&v)
+ } else {
+ parse_subreddit_about(&v)
+ }
+}
+
+fn parse_subreddit_about(v: &serde_json::Value) -> Option<EntityData> {
+ let data = v.get("data")?;
+ let title = data
+ .get("title")
+ .or_else(|| data.get("display_name"))
+ .and_then(|t| t.as_str())?
+ .to_string();
+ let body_html = data
+ .get("public_description_html")
+ .or_else(|| data.get("public_description"))
+ .and_then(|t| t.as_str())
+ .map(|s| s.to_string());
+ let thumb_url = data
+ .get("icon_img")
+ .or_else(|| data.get("community_icon"))
+ .and_then(|t| t.as_str())
+ .filter(|s| !s.is_empty())
+ .map(|s| s.to_string());
+
+ Some(EntityData {
+ title,
+ author: None,
+ body_html,
+ thumb_url,
+ })
+}
+
+fn parse_post_listing(v:
… preview truncated; 4,570 characters omittedHardlinks — judgments / attempts / prompt
judgments
attempts
Prompt text is loaded only by the download route.