Side B fixes a real production bug (Reddit blocking cloud IPs via public API fallback) with a robust auth-rejection/retry mechanism, better error messages, and a config pin, backed by a new test. Side A is mostly a refactor/removal of demo scaffolding plus a settlement-worker rewrite for ranking caching, which is reasonable engineering but less impactful and riskier (async batching complexity) without clear bug motivation.
constitution · epochs · watch · epoch 3
comparison
c_4772ee88dbe3 (tommy-mor) vs c_ca9169f732b8 (tommy-mor)
download prompt · raw event · cmp_b48704150b67c4
council reasoning
A introduces a real settlement worker (batched votes, disk append, off-request ranking recompute), a cached ranked_items fast path with read locks, and deletes the demo-counter scaffold end-to-end—core architecture for the ranking product. B is a solid, narrower production fix (force OAuth when creds exist, refresh on 401/403, error truncation, fly base URL) but does not reshape the system the way A does.
Side A introduces a substantial architectural improvement by separating ranking recomputation from read paths via cached rankings, adding a settlement worker that batches vote persistence and score recomputation, warming the cache at startup, and converting ranking reads to shared locks. It also removes the temporary demo counter feature and associated UI/tests, reducing dead functionality. Side B fixes an important Reddit OAuth reliability issue by requiring OAuth when configured, refreshing tokens on 401/403, and improving error handling, but its impact is confined to one subsystem and is less broadly foundational than the state-management and performance changes in Side A.
sides
A — c_4772ee88dbe3 (tommy-mor)
message
[07715165] nice
diff preview
diff --git a/server/src/api/ui_html.rs b/server/src/api/ui_html.rs
index c3c62a76f424010d77a6090c84dd0b82098f573e..da2536112faea313352624cf2ce0ddd0ab3377c1 100644
--- a/server/src/api/ui_html.rs
+++ b/server/src/api/ui_html.rs
@@ -6,7 +6,7 @@ use axum::{
use std::collections::HashMap;
use crate::{
- html::{demo_counter_panel, js_string_literal, ranking_panel, JsBuilder},
+ html::{js_string_literal, ranking_panel, JsBuilder},
parser::parse_reddit_url,
parser_render::parser_panel_morph,
state::AppState,
@@ -35,13 +35,6 @@ pub async fn post_ui_html(
};
match action {
- HtmlUiAction::BumpDemoCounter => {
- let count = state.bump_demo_counter().await;
- let panel = demo_counter_panel(count, state.event_log.path().to_string_lossy().as_ref());
- JsBuilder::new()
- .morph_selector("#demo-counter-panel", panel)
- .into_response()
- }
HtmlUiAction::RecordVote {
a,
b,
@@ -54,8 +47,8 @@ pub async fn post_ui_html(
{
return ui_js_warn(&e).into_response();
}
- let mut group = state.group.write().await;
- let panel = ranking_panel(&mut group);
+ let group = state.group.read().await;
+ let panel = ranking_panel(&group);
JsBuilder::new()
.morph_selector("#ranking-panel", panel)
.into_response()
@@ -87,20 +80,6 @@ mod tests {
assert!(matches!(err, HtmlUiParseError::MissingRpc));
}
- #[test]
- fn bump_action_deserializes() {
- let template = serde_json::json!({ "action": "bump_demo_counter" });
- let mut form = HashMap::new();
- form.insert(
- UI_RPC_FIELD.to_string(),
- serde_json::to_string(&template).unwrap(),
- );
- assert_eq!(
- parse_html_ui_from_form(&form).unwrap(),
- HtmlUiAction::BumpDemoCounter
- );
- }
-
#[test]
fn record_vote_action_deserializes() {
let template = serde_json::json!({
diff --git a/server/src/events.rs b/server/src/events.rs
index b969242534e184d4f0a689543a479670b08a18df..eff80aef0257f706d2341f666e63d6a3d921bf6e 100644
--- a/server/src/events.rs
+++ b/server/src/events.rs
@@ -5,8 +5,6 @@ use serde::{Deserialize, Serialize};
pub enum Event {
/// Page view recorded (path → counter in views.json).
ViewRecorded { path: String, ts: i64 },
- /// Demo counter bump from `POST /ui` (persisted in the single JSONL log).
- DemoCounterBumped { ts: i64, value: u64 },
/// Pairwise comparison vote (replayed into [`crate::reducer::GroupState`] on boot).
VoteRecorded {
ts: i64,
diff --git a/server/src/html/mod.rs b/server/src/html/mod.rs
index 5b1d0b5a887d89e7e80796aa7a6c8ed5baaf2782..d69ed962b5c8625bc83c933b1825f2cc1d0868e2 100644
--- a/server/src/html/mod.rs
+++ b/server/src/html/mod.rs
@@ -13,7 +13,7 @@ use crate::{
form_template::template_json_compact,
parser_action::ParserAction,
parser_render::parser_panel,
- ranking::ranked_items,
+ ranking::ranked_items_cached,
reducer::GroupState,
state::AppState,
ui_action::UI_RPC_FIELD,
@@ -199,10 +199,8 @@ fn layout(title: &str, body: Markup, views: u64, theme: &str, theme_next: &str)
}
}
-pub fn ranking_panel(group: &mut GroupState) -> Markup {
- const MAX_ITERS: usize = 10_000;
- const TOL: f64 = 1e-8;
- let items = ranked_items(group, MAX_ITERS, TOL);
+pub fn ranking_panel(group: &GroupState) -> Markup {
+ let items = ranked_items_cached(group);
html! {
section id="ranking-panel" class="demo-panel" {
h2 { "Ranking" }
@@ -260,35 +258,6 @@ pub fn vote_panel() -> Markup {
}
-pub fn demo_counter_panel(count: u64, event_log_path: &str) -> Markup {
- let rpc = template_json_compact(&serde_json::json!({ "action": "bump_demo_counter" }))
- .expect("rpc json");
- html! {
- section id="demo-counter-panel" class="demo-panel" {
- h1 { "sorter2" }
- p class="muted" {
- "Pairwise ranking scaffold — votes persist to JSONL and replay on boot."
- }
- p class="demo-count" {
- strong { "Counter: " }
- span id="demo-count-value" { (count) }
- }
- p class="muted small" {
- "Event log: " code { (event_log_path) }
- }
- form method="post" action="/ui" id="demo-bump-form" {
- input type="hidden" name=(UI_RPC_FIELD) value=(rpc);
- button type="submit" class="btn-primary" { "Bump (POST /ui → eval JS)" }
- }
- p class="muted small" {
- "Uses hidden "
- code { "__rpc__" }
- " JSON + Idiomorph morph — no full page reload."
- }
- }
- }
-}
-
pub async fn home(
State(state): State<AppState>,
jar: CookieJar,
@@ -297,16 +266,15 @@ pub async fn home(
let path = uri.path().to_string();
state.views.increment(path.clone());
let views = state.views.get_views(&path);
- let count = *state.demo_counter.read().await;
let theme = theme_from_jar(&jar);
let theme_next = theme_next_from_uri(&uri);
- let mut group = state.group.write().await;
+ let group = state.group.read().await;
let empty_action = ParserAction::suggest(String::new(), None);
let body = html! {
+ h1 { "sorter2" }
(parser_panel("", &empty_action))
(vote_panel())
- (ranking_panel(&mut group))
- (demo_counter_panel(count, state.event_log.path().to_string_lossy().as_ref()))
+ (ranking_panel(&group))
};
layout("sorter2", body, views, theme, &theme_next)
}
diff --git a/server/src/lib.rs b/server/src/lib.rs
index 6716c5b282e7980a7a0f03d63ad8b25eda61cc55..fa423640d598f4ba97a5885d228e78d7b97f7a22 100644
--- a/server/src/lib.rs
+++ b/server/src/lib.rs
@@ -9,6 +9,7 @@ pub mod parser_render;
pub mod path_types;
pub mod ranking;
pub mod reducer;
+pub mod settlement;
pub mod state;
pub mod ui_action;
pub mod views;
diff --git a/server/src/ranking.rs b/server/src/ranking.rs
index 89d3280126a8d8f841721ce8cb63ff735d68752a..2d706762792ba9239bb3f1c2e4974a2fde908013 100644
--- a/server/src/ranking.rs
+++ b/server/src/ranking.rs
@@ -91,6 +91,11 @@ pub fn compute_group_ranking(group: &mut GroupState, max_iters: usize, tol: f64)
pub fn ranked_items(group: &mut GroupState, max_iters: usize, tol: f64) -> Vec<RankedItem> {
compute_group_ranking(group, max_iters, tol);
+ ranked_items_cached(group)
+}
+
+/// Read cached scores without recomputing (HTTP fast path).
+pub fn ranked_items_cached(group: &GroupState) -> Vec<RankedItem> {
let mut items: Vec<RankedItem> = group
.idx_to_item
.iter()
@@ -105,7 +110,12 @@ pub fn ranked_items(group: &mut GroupState, max_iters: usize, tol: f64) -> Vec<R
items
}
-fn compute_scores_from_edges(n: usize, edges: impl Iterator<Item = ((usize, usize), f64)>, max_iters: usize, tol: f64) -> Vec<f64> {
+pub fn compute_scores_from_edges(
+ n: usize,
+ edges: impl Iterator<Item = ((usize, usize), f64)>,
+ max_iters: usize,
+ tol: f64,
+) -> Vec<f64> {
if n == 0 {
return vec![];
}
diff --git a/server/src/settlement.rs b/server/src/settlement.rs
new file mode 100644
index 0000000000000000000000000000000000000000..1f722ceaea62cda22c28ab71551f259fbf049b81
--- /dev/null
+++ b/server/src/settlement.rs
@@ -0,0 +1,114 @@
+use std::sync::Arc;
+
+use tokio::sync::{mpsc, oneshot, RwLock};
+
+use crate::{
+ event_log::EventLog,
+ events::Event,
+ ranking::compute_scores_from_edges,
+ reducer::{GroupState, VoteData},
+};
+
+const MAX_ITERS: usize = 10_000;
+const TOL: f64 = 1e-8;
+
+pub struct SettlementCommand {
+ pub vote: VoteData,
+ pub event: Event,
+ pub reply: oneshot::Sender<Result<(), String>>,
+}
+
+#[derive(Clone)]
+pub struct SettlementClient {
+ tx: mpsc::Sender<SettlementCommand>,
+}
+
+impl SettlementClient {
+ pub fn spawn(group: Arc<RwLock<GroupState>>, event_log: Arc<EventLog>) -> Self {
+ let (tx, rx) = mpsc::channel(64);
+ tokio::spawn(settlement_worker(rx, group, event_log));
+ Self { tx }
+ }
+
+ pub async fn record_vote(&self, vote: VoteData, event: Event) -> Result<(), String> {
+ let (reply, rx) = oneshot::channel();
+ self.tx
+ .send(SettlementCommand {
+ vote,
+ event,
+ reply,
+ })
+ .await
+ .map_err(|_| "settlement worker stopped".to_string())?;
+ rx.await
+ .map_err(|_| "settlement worker stopped".to_string())?
+ }
+}
+
+async fn settlement_worker(
+ mut rx: mpsc::Receiver<SettlementCommand>,
+ group: Arc<RwLock<GroupState>>,
+ event_log: Arc<EventLog>,
+) {
+ while let Some(first) = rx.recv().await {
+ let mut batch = vec![first];
+ while let Ok(more) = rx.try_recv() {
+ batch.push(more);
+ }
+
+ let mut disk_err: Option<String> = None;
+ for cmd in &batch {
+ if let Err(e) = event_log.append(&cmd.event).await {
+ disk_err = Some(e.to_string());
+ break;
+ }
+ }
+
+ if let Some(err) = disk_err {
+ for cmd in batch {
+ let _ = cmd.reply.send(Err(err.clone()));
+ }
+ continue;
+ }
+
+ let (edges, n) = {
+ let mut w = group.write().await;
+ for cmd in &batch {
+ w.apply_vote(cmd.vote.clone());
+ }
+ (w.edges.clone(), w.idx_to_item.len())
+ };
+
+ let new_scores = compute_scores_from_edges(
+ n,
+ edges.iter().map(|(&k, &v)| (k, v)),
+ MAX_ITERS,
+ TOL,
+ );
+
+ {
+ let mut w = group.write().await;
+ w.cached_scores = new_scores;
+ w.dirty = false;
+ }
+
+ for cmd in batch {
+ let _ = cmd.reply.send(Ok(()));
+ }
+ }
+}
+
+/// Compute ranking cache from current in-memory edges (startup replay only).
+pub fn warm_ranking_cache(group: &mut GroupState) {
+ if !group.dirty {
+ return;
+ }
+ let n = group.idx_to_item.len();
+ group.cached_scores = compute_scores_from_edges(
+ n,
+ group.edges.iter().map(|(&k, &v)| (k, v)),
+ MAX_ITERS,
+ TOL,
+ );
+ group.dirty = false;
+}
diff --git a/server/src/state.rs b/server/src/state.rs
index 8ec9902e2ecc31cf8208f7ad6365891dc5537eed..1922541a4064c2de1df2d993a461cae783320e05 100644
--- a/server/src/state.rs
+++ b/server/src/state.rs
@@ -6,6 +6,7 @@ use crate::{
event_log::EventLog,
events::Event,
reducer::{GroupState, VoteData},
+ settlement::{warm_ranking_cache, SettlementClient},
views::ViewStore,
};
@@ -38,8 +39,8 @@ pub struct AppState {
pub cfg: Arc<AppConfig>,
pub event_log: Arc<EventLog>,
pub views: ViewStore,
- pub demo_counter: Arc<RwLock<u64>>,
pub group: Arc<RwLock<GroupState>>,
+ settlement: SettlementClient,
}
impl AppState {
@@ -48,14 +49,10 @@ impl AppState {
let views_path = format!("{}/views.json", cfg.data_dir);
let views = ViewStore::new(&views_path);
- let mut demo_counter: u64 = 0;
let mut group = GroupState::new();
if let Ok((events, _)) = event_log.load_all().await {
for ev in events {
match ev {
- Event::DemoCounterBumped { value, .. } => {
- demo_counter = demo_counter.max(value);
- }
Event::VoteRecorded {
ts,
a,
@@ -74,30 +71,20 @@ impl AppState {
}
}
+
… preview truncated; 6,096 characters omittedB — c_ca9169f732b8 (tommy-mor)
message
[8f69c309] Require Reddit OAuth when credentials are set and refresh on 401/403. Avoid falling back to the public www.reddit.com API from cloud IPs, which returns Reddit's network-security block page. Also pin SORTER2_BASE_URL in fly.toml. Co-authored-by: Cursor <cursoragent@cursor.com>
diff preview
diff --git a/fly.toml b/fly.toml
index f0c6a39f643c204987debc177234d95a8ef44b65..ca7e0088a7efd7d58d29d808f8d89080b2bff233 100644
--- a/fly.toml
+++ b/fly.toml
@@ -5,6 +5,7 @@ primary_region = "iad"
dockerfile = "Dockerfile"
[env]
+ SORTER2_BASE_URL = "https://reddit.sorter.social"
SORTER2_DATA_DIR = "/data"
SORTER2_EVENT_LOG = "/data/events.jsonl"
PORT = "8080"
diff --git a/server/src/reddit.rs b/server/src/reddit.rs
index a874814f8927192ee62cab2d0db1efd27dcd57b7..f409764c1e1f36216f1b08107043c2eab905694c 100644
--- a/server/src/reddit.rs
+++ b/server/src/reddit.rs
@@ -283,26 +283,35 @@ async fn reddit_worker(
);
tokio::time::sleep(current_delay).await;
- if let Some(c) = &creds {
- oauth = ensure_oauth_token(&client, &oauth_token_base, c, oauth.take()).await;
- }
-
- let token = oauth.as_ref().map(|t| t.access_token.as_str());
- let fetch_base = if token.is_some() {
- tracing::debug!(
- item = %fetch_id,
- base = %oauth_api_base,
- "reddit fetch using OAuth bearer"
- );
- &oauth_api_base
- } else {
- &api_base
- };
- let url = match kind {
- FetchKind::SelfEntity => map_item_to_reddit_api(&fetch_id, fetch_base),
- FetchKind::Children => map_children_url(&fetch_id, fetch_base),
+ let outcome = match &creds {
+ Some(c) => {
+ // OAuth is required when credentials are configured — never fall
+ // back to the public www.reddit.com JSON endpoints (cloud IPs
+ // get blocked with a 403 HTML interstitial).
+ fetch_with_oauth(
+ &client,
+ &oauth_token_base,
+ &oauth_api_base,
+ c,
+ &mut oauth,
+ &fetch_id,
+ kind,
+ )
+ .await
+ }
+ None => {
+ let url = match kind {
+ FetchKind::SelfEntity => map_item_to_reddit_api(&fetch_id, &api_base),
+ FetchKind::Children => map_children_url(&fetch_id, &api_base),
+ };
+ match do_fetch(&client, &url, &fetch_id, None).await {
+ Ok(FetchOutcome::AuthRejected { status, detail }) => {
+ Err(format!("Reddit API {status}: {detail}"))
+ }
+ other => other,
+ }
+ }
};
- let outcome = do_fetch(&client, &url, &fetch_id, token).await;
match outcome {
Ok(FetchOutcome::Payload(payload)) => {
@@ -342,6 +351,12 @@ async fn reddit_worker(
current_delay = (current_delay * 2).min(Duration::from_secs(60));
notify(done, FetchJobResult::RateLimited { reset_secs });
}
+ Ok(FetchOutcome::AuthRejected { status, detail }) => {
+ let e = format!("Reddit API {status}: {detail}");
+ tracing::warn!(item = %fetch_id, err = %e, "reddit fetch auth rejected");
+ current_delay = (current_delay * 2).min(Duration::from_secs(60));
+ notify(done, FetchJobResult::Failed(e));
+ }
Err(e) => {
tracing::warn!(item = %fetch_id, err = %e, "reddit fetch failed");
current_delay = (current_delay * 2).min(Duration::from_secs(60));
@@ -357,6 +372,60 @@ enum FetchOutcome {
Payload(Value),
NotFound,
RateLimited { reset_secs: u64 },
+ /// Bearer rejected — caller should drop the cached token and retry once.
+ AuthRejected { status: StatusCode, detail: String },
+}
+
+async fn fetch_with_oauth(
+ client: &Client,
+ oauth_token_base: &str,
+ oauth_api_base: &str,
+ creds: &RedditCredentials,
+ oauth: &mut Option<OAuthToken>,
+ fetch_id: &ItemId,
+ kind: FetchKind,
+) -> Result<FetchOutcome, String> {
+ for attempt in 0..2 {
+ let force_refresh = attempt > 0;
+ *oauth = Some(
+ ensure_oauth_token(client, oauth_token_base, creds, oauth.take(), force_refresh)
+ .await?,
+ );
+ let token = oauth
+ .as_ref()
+ .expect("token set above")
+ .access_token
+ .clone();
+
+ tracing::debug!(
+ item = %fetch_id,
+ base = %oauth_api_base,
+ attempt,
+ "reddit fetch using OAuth bearer"
+ );
+
+ let url = match kind {
+ FetchKind::SelfEntity => map_item_to_reddit_api(fetch_id, oauth_api_base),
+ FetchKind::Children => map_children_url(fetch_id, oauth_api_base),
+ };
+ match do_fetch(client, &url, fetch_id, Some(&token)).await? {
+ FetchOutcome::AuthRejected { status, detail } if attempt == 0 => {
+ tracing::warn!(
+ item = %fetch_id,
+ %status,
+ %detail,
+ "reddit OAuth rejected; refreshing token and retrying"
+ );
+ *oauth = None;
+ continue;
+ }
+ FetchOutcome::AuthRejected { status, detail } => {
+ return Err(format!("Reddit API {status}: {detail}"));
+ }
+ other => return Ok(other),
+ }
+ }
+ unreachable!("loop always returns")
}
async fn ensure_oauth_token(
@@ -364,35 +433,35 @@ async fn ensure_oauth_token(
oauth_base: &str,
creds: &RedditCredentials,
existing: Option<OAuthToken>,
-) -> Option<OAuthToken> {
- if let Some(t) = existing {
- if Instant::now() < t.expires_at - Duration::from_secs(60) {
- tracing::debug!("reddit OAuth token still valid");
- return Some(t);
+ force_refresh: bool,
+) -> Result<OAuthToken, String> {
+ if !force_refresh {
+ if let Some(t) = existing {
+ if Instant::now() < t.expires_at - Duration::from_secs(60) {
+ tracing::debug!("reddit OAuth token still valid");
+ return Ok(t);
+ }
}
}
let url = format!("{}/api/v1/access_token", oauth_base.trim_end_matches('/'));
- tracing::debug!(%url, "reddit OAuth token request");
+ tracing::debug!(%url, force_refresh, "reddit OAuth token request");
let resp = client
.post(&url)
.basic_auth(&creds.client_id, Some(&creds.client_secret))
.form(&[("grant_type", "client_credentials")])
.send()
- .await;
-
- let resp = match resp {
- Ok(r) => r,
- Err(e) => {
- tracing::warn!("reddit OAuth token request failed: {e}");
- return None;
- }
- };
+ .await
+ .map_err(|e| format!("Reddit OAuth token request failed: {e}"))?;
if !resp.status().is_success() {
- tracing::warn!("reddit OAuth token HTTP {}", resp.status());
- return None;
+ let status = resp.status();
+ let body = resp.text().await.unwrap_or_default();
+ return Err(format!(
+ "Reddit OAuth token HTTP {status}: {}",
+ truncate_for_error(&body)
+ ));
}
#[derive(Deserialize)]
@@ -401,21 +470,40 @@ async fn ensure_oauth_token(
expires_in: u64,
}
- let body: TokenResponse = match resp.json().await {
- Ok(b) => b,
- Err(e) => {
- tracing::warn!("reddit OAuth token parse failed: {e}");
- return None;
- }
- };
+ let body: TokenResponse = resp
+ .json()
+ .await
+ .map_err(|e| format!("Reddit OAuth token parse failed: {e}"))?;
- tracing::debug!(expires_in = body.expires_in, "reddit OAuth token acquired");
- Some(OAuthToken {
+ tracing::info!(expires_in = body.expires_in, "reddit OAuth token acquired");
+ Ok(OAuthToken {
access_token: body.access_token,
expires_at: Instant::now() + Duration::from_secs(body.expires_in),
})
}
+fn truncate_for_error(body: &str) -> String {
+ let compact: String = body.split_whitespace().collect::<Vec<_>>().join(" ");
+ if compact.is_empty() {
+ return "(empty body)".into();
+ }
+ // Prefer the human-readable block message over dumping Reddit's CSS.
+ if let Some(idx) = compact.find("You've been blocked") {
+ let slice: String = compact.chars().skip(idx).take(160).collect();
+ return if compact.chars().count() > idx + 160 {
+ format!("{slice}…")
+ } else {
+ slice
+ };
+ }
+ let chars: String = compact.chars().take(200).collect();
+ if compact.chars().count() > 200 {
+ format!("{chars}…")
+ } else {
+ chars
+ }
+}
+
async fn do_fetch(
client: &Client,
url: &str,
@@ -460,15 +548,16 @@ async fn do_fetch(
if !status.is_success() {
let body = resp.text().await.unwrap_or_default();
+ let detail = truncate_for_error(&body);
tracing::debug!(
item = %id,
%status,
body_len = body.len(),
- body_prefix = %body.chars().take(240).collect::<String>(),
+ %detail,
"reddit non-success body"
);
if status == StatusCode::FORBIDDEN || status == StatusCode::UNAUTHORIZED {
- return Err(format!("Reddit API {status}: {body}"));
+ return Ok(FetchOutcome::AuthRejected { status, detail });
}
return Ok(FetchOutcome::NotFound);
}
@@ -716,6 +805,15 @@ fn reddit_direct_image_url(url: &str) -> bool {
mod tests {
use super::*;
+ #[test]
+ fn truncate_error_prefers_block_message() {
+ let html = r#"<style>.x{color:red}</style><div>You've been blocked by network security. To continue, log in</div>"#;
+ let msg = truncate_for_error(html);
+ assert!(msg.starts_with("You've been blocked"));
+ assert!(msg.len() < 200);
+ assert!(!msg.contains(".x{color"));
+ }
+
#[test]
fn map_subreddit_about_url() {
let id = ItemId::from_url("https://reddit.com/r/rust").unwrap();
Hardlinks — judgments / attempts / prompt
judgments
attempts
Prompt text is loaded only by the download route.