Side B is a substantial architectural change that removes an entire subsystem (EntityStore), redefines the event log semantics to keep Reddit content ephemeral with TTL eviction, and touches storage schema, journal, projection, reducer, state, and integration tests consistently across the codebase—delivering real durability/privacy/lasting-value improvements. Side A is a small, well-scoped bugfix improving a color-gradient heuristic in one function with good tests, but it is narrow in scope and impact compared to B's systemic redesign.
constitution · epochs · watch · epoch 3
comparison
c_66eb04076a98 (tommy-mor) vs c_5cd3e5917d2f (tommy-mor)
download prompt · raw event · cmp_debbb905e4cbdc
council reasoning
B removes EntityImported/EntityStore entirely, keeps only NodeEnsured in the durable log, and adds TTL eviction for ephemeral Reddit display content—fixing retention policy, log bloat, and rebuild semantics across journal, projection, storage schema, and reddit import. A only retunes rank-row gradient math from list ordinal to per-group min–max score, a correct but narrow UI polish.
Side B makes a substantial architectural change: it removes the persistent EntityStore and EntityImported event, moves Reddit display data into ephemeral projection storage with TTL-based eviction, updates replay/journaling paths accordingly, and adds tests verifying rebuild and eviction behavior. Side A is a well-scoped UI improvement that changes row color gradients from list position to per-group score normalization with supporting tests, but its impact is limited to presentation rather than core system design and data handling.
sides
A — c_66eb04076a98 (tommy-mor)
message
[0366806e] Color rank rows by vote mass within each group, not list position. Min–max normalization keeps similar scores visually close while still using the full gradient as groups grow and absolute mass dilutes. Co-authored-by: Cursor <cursoragent@cursor.com>
diff preview
diff --git a/server/src/html/mod.rs b/server/src/html/mod.rs
index a58cbbee3490a08a625cb06df06848c59a615d65..4eff2e19ed4d303ff8e80c1eabd8a15b4990e643 100644
--- a/server/src/html/mod.rs
+++ b/server/src/html/mod.rs
@@ -264,12 +264,19 @@ pub fn scope_theme_style(parent: &ItemId) -> String {
)
}
-fn rank_row_style(parent: &ItemId, ordinal: usize, total: usize) -> String {
- let t = if total <= 1 {
- 0.0
- } else {
- ordinal as f64 / (total - 1) as f64
- };
+/// Map vote mass to gradient position using the group's score range, not raw mass or
+/// list position. Vote mass sums to 1 across the component, so absolute values dilute
+/// as N grows; min–max within the visible list preserves similar scores → similar colors.
+fn score_gradient_t(score: f64, min_score: f64, max_score: f64) -> f64 {
+ let spread = max_score - min_score;
+ if spread < 1e-9 {
+ return 0.5;
+ }
+ ((max_score - score) / spread).clamp(0.0, 1.0)
+}
+
+fn rank_row_style(parent: &ItemId, score: f64, min_score: f64, max_score: f64) -> String {
+ let t = score_gradient_t(score, min_score, max_score);
let base_hue = scope_base_hue(parent);
let hue = (base_hue + 118.0 * t) % 360.0;
let lightness = 0.74 - 0.34 * t;
@@ -295,14 +302,15 @@ fn rank_list(
highlighted: &HashSet<ItemId>,
tree: &GlobalTree,
) -> Markup {
- let group_len = items.len();
+ let min_score = items.iter().map(|r| r.score).fold(f64::INFINITY, f64::min);
+ let max_score = items.iter().map(|r| r.score).fold(f64::NEG_INFINITY, f64::max);
html! {
@if !items.is_empty() {
h3 class="rank-heading muted small" { (label) }
ol class="rank-list" {
@for (i, r) in items.iter().enumerate() {
@let href = item_href(&r.item);
- @let style = rank_row_style(parent, i, group_len);
+ @let style = rank_row_style(parent, r.score, min_score, max_score);
@let class = rank_row_class(&r.item, highlighted);
li class=(class)
data-rank-item=(r.item.as_str())
@@ -517,19 +525,37 @@ pub async fn browse(State(state): State<AppState>, uri: Uri) -> impl IntoRespons
#[cfg(test)]
mod tests {
- use super::{rank_row_style, SORTER_UI_JS};
+ use super::{rank_row_style, score_gradient_t, SORTER_UI_JS};
use crate::path_types::ItemId;
#[test]
- fn rank_row_style_gradients_per_group_not_globally() {
+ fn score_gradient_t_uses_group_range_not_absolute_mass() {
+ assert!((score_gradient_t(0.12, 0.08, 0.12) - 0.0).abs() < 1e-9);
+ assert!((score_gradient_t(0.08, 0.08, 0.12) - 1.0).abs() < 1e-9);
+ // Raw 12% mass would map near the dark end globally; within this group it's the top.
+ assert!(score_gradient_t(0.12, 0.08, 0.12) < score_gradient_t(0.12, 0.0, 1.0));
+ }
+
+ #[test]
+ fn score_gradient_t_similar_scores_similar_t() {
+ let a = score_gradient_t(0.41, 0.20, 0.60);
+ let b = score_gradient_t(0.40, 0.20, 0.60);
+ assert!((a - b).abs() < 0.05);
+ assert!((a - score_gradient_t(0.60, 0.20, 0.60)).abs() > 0.3);
+ }
+
+ #[test]
+ fn score_gradient_t_tied_scores_neutral() {
+ assert!((score_gradient_t(0.25, 0.25, 0.25) - 0.5).abs() < 1e-9);
+ }
+
+ #[test]
+ fn rank_row_style_same_inputs_same_color() {
let parent = ItemId::opaque("test-scope");
- let first_in_four = rank_row_style(&parent, 0, 4);
- let last_in_four = rank_row_style(&parent, 3, 4);
- let first_in_two = rank_row_style(&parent, 0, 2);
- let last_in_two = rank_row_style(&parent, 1, 2);
- assert_eq!(first_in_four, first_in_two);
- assert_eq!(last_in_four, last_in_two);
- assert_ne!(first_in_four, last_in_four);
+ assert_eq!(
+ rank_row_style(&parent, 0.33, 0.20, 0.60),
+ rank_row_style(&parent, 0.33, 0.20, 0.60),
+ );
}
#[test]
B — c_5cd3e5917d2f (tommy-mor)
message
[1d14ff09] Ephemeral Reddit content; log structure only (#46) * Keep Reddit content ephemeral; log structure only Remove EntityImported and EntityStore. Reddit fetches write display content directly to the projection with a fetched_at timestamp, while the event log records NodeEnsured for discovered identities only. A background task evicts cached display content after 48 hours. Votes, tree structure, and ItemIds remain in the log and projection. Co-authored-by: tommy <thmorriss@gmail.com> * Fix reddit import test assertions and Clojure syntax Co-authored-by: tommy <thmorriss@gmail.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
diff preview
diff --git a/server/src/bin/storage_bench.rs b/server/src/bin/storage_bench.rs
index 4762d0d23bc8e2df439afe09a3235ba7f72bd486..3d988416ad36b27a7d3dc84280cfdbdcafa43e69 100644
--- a/server/src/bin/storage_bench.rs
+++ b/server/src/bin/storage_bench.rs
@@ -6,8 +6,8 @@ use std::{
};
use sorter2_server::{
- entity_store::EntityStore, event_log::EventLog, events::Event, journal::JournalClient,
- projection_apply, projection_store::ProjectionStore,
+ event_log::EventLog, events::Event, journal::JournalClient, projection_apply,
+ projection_store::ProjectionStore,
};
#[tokio::main]
@@ -18,12 +18,10 @@ async fn main() -> Result<(), Box<dyn Error + Send + Sync>> {
let data_dir = opts.data_dir.to_string_lossy().into_owned();
let event_log = Arc::new(EventLog::new(format!("{data_dir}/events.jsonl")));
let db = durable::Db::open(opts.data_dir.join("store"))?;
- let entity_store = EntityStore::from_db(&db)?;
let projection_store = ProjectionStore::from_db(&db)?;
let journal = JournalClient::spawn(
event_log.clone(),
- entity_store.clone(),
projection_store.clone(),
event_log.last_sequence().await? + 1,
);
@@ -46,12 +44,9 @@ async fn main() -> Result<(), Box<dyn Error + Send + Sync>> {
drop(journal);
let rebuild_start = Instant::now();
- entity_store.reset()?;
projection_store.reset()?;
let rebuild = event_log
- .replay(|record| {
- projection_apply::apply_records(&projection_store, &entity_store, &[record])
- })
+ .replay(|record| projection_apply::apply_records(&projection_store, &[record]))
.await?;
let rebuild_elapsed = rebuild_start.elapsed();
diff --git a/server/src/entity_store.rs b/server/src/entity_store.rs
deleted file mode 100644
index d29f39aecca6f12cdcf263cf77c3654eb4ee6cfa..0000000000000000000000000000000000000000
--- a/server/src/entity_store.rs
+++ /dev/null
@@ -1,134 +0,0 @@
-//! Off-heap storage for full entity payloads (Reddit API JSON).
-//!
-//! Derived [`crate::reducer::EntityData`] is stored on the node; the raw JSON
-//! lives here, in the shared durable [`Store`] schema.
-
-use std::path::Path;
-
-use durable::{Batch, Db, Durability};
-use serde_json::Value;
-
-use crate::{
- path_types::ItemId,
- storage_dto::{decode_entity_payload, encode_entity_payload},
- storage_schema::{Store, StoreFields},
-};
-
-const ENTITY_SCHEMA_KEY: &str = "schema_version";
-const ENTITY_SCHEMA_VERSION: u64 = 2;
-
-#[derive(Debug, thiserror::Error)]
-pub enum EntityStoreError {
- #[error("durable error: {0}")]
- Durable(#[from] durable::Error),
- #[error("json error: {0}")]
- Json(#[from] serde_json::Error),
- #[error("storage decode error: {0}")]
- Storage(String),
- #[error("io error: {0}")]
- Io(#[from] std::io::Error),
-}
-
-/// Disk-backed map of entity id → raw JSON payload.
-#[derive(Clone)]
-pub struct EntityStore {
- db: Db,
-}
-
-impl EntityStore {
- /// Open (or create) the entity database under `dir`.
- pub fn open(dir: &Path) -> Result<Self, EntityStoreError> {
- std::fs::create_dir_all(dir)?;
- let db = Db::open(dir)?;
- Self::from_db(&db)
- }
-
- /// Create an entity store backed by an already-open database.
- pub fn from_db(db: &Db) -> Result<Self, EntityStoreError> {
- let store = Self { db: db.clone() };
- let version = Store::root()
- .entity_meta()
- .key(&ENTITY_SCHEMA_KEY.to_string())
- .get(db)?;
- if version != Some(ENTITY_SCHEMA_VERSION) {
- store.reset()?;
- }
- Ok(store)
- }
-
- /// Clear rebuildable entity payloads and reset storage schema metadata.
- pub fn reset(&self) -> Result<(), EntityStoreError> {
- let root = Store::root();
- self.db.apply(
- &[root.entities().clear(), root.entity_meta().clear()],
- Durability::SyncWal,
- )?;
- self.db.run(
- root.entity_meta()
- .key(&ENTITY_SCHEMA_KEY.to_string())
- .set(&ENTITY_SCHEMA_VERSION),
- Durability::SyncWal,
- )?;
- Ok(())
- }
-
- /// Persist a payload for `id` (overwrites any existing entry).
- pub fn put(&self, id: &ItemId, payload: &Value) -> Result<(), EntityStoreError> {
- self.db.run(
- Store::root()
- .entities()
- .key(&id.as_str().to_string())
- .set(&encode_entity_payload(payload)),
- Durability::SyncWal,
- )?;
- Ok(())
- }
-
- /// Add a payload write to the caller's batch.
- pub fn put_in_batch(
- &self,
- batch: &mut Batch,
- id: &ItemId,
- payload: &Value,
- ) -> Result<(), EntityStoreError> {
- batch.write(
- Store::root()
- .entities()
- .key(&id.as_str().to_string())
- .set(&encode_entity_payload(payload)),
- );
- Ok(())
- }
-
- /// Load a stored payload, if present.
- pub fn get(&self, id: &ItemId) -> Result<Option<Value>, EntityStoreError> {
- match Store::root()
- .entities()
- .key(&id.as_str().to_string())
- .get(&self.db)?
- {
- Some(record) => decode_entity_payload(record)
- .map(Some)
- .map_err(EntityStoreError::Storage),
- None => Ok(None),
- }
- }
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
- use serde_json::json;
-
- #[test]
- fn round_trip_payload() {
- let tmp = tempfile::tempdir().unwrap();
- let store = EntityStore::open(tmp.path()).unwrap();
- let id = ItemId::from_url("https://reddit.com/r/rust").unwrap();
- let payload = json!({"kind": "t5", "data": {"display_name": "rust"}});
-
- store.put(&id, &payload).unwrap();
- let loaded = store.get(&id).unwrap().unwrap();
- assert_eq!(loaded, payload);
- }
-}
diff --git a/server/src/events.rs b/server/src/events.rs
index a3d88e285c645a96d33e6f0aed1b487b843aceb8..d76c3bb4277216b0d39c9422ba7a50db10a95e05 100644
--- a/server/src/events.rs
+++ b/server/src/events.rs
@@ -1,5 +1,4 @@
use serde::{Deserialize, Serialize};
-use serde_json::Value;
/// Schema version for JSONL log records. Bump when event semantics change.
pub const CURRENT_LOG_SCHEMA: u32 = 1;
@@ -30,7 +29,7 @@ pub type ViewRecord = LogRecord<ViewEvent>;
/// Wall-clock timestamp carried on the log envelope for domain events.
pub fn event_timestamp(event: &Event) -> i64 {
match event {
- Event::VoteRecorded { ts, .. } | Event::EntityImported { ts, .. } => *ts,
+ Event::VoteRecorded { ts, .. } => *ts,
Event::NodeEnsured { .. } => crate::fetch::now_ms(),
}
}
@@ -57,6 +56,4 @@ pub enum Event {
},
/// Register a node path in the fractal tree (no external fetch).
NodeEnsured { id: String },
- /// Full upstream API payload for a node (domain-specific view derived at replay/render time).
- EntityImported { id: String, ts: i64, payload: Value },
}
diff --git a/server/src/journal.rs b/server/src/journal.rs
index 50bc89f976edb82b7b0e49e954a8eccbbe82bf87..d50023aca7c3e74068de988449b11baee32d1ecf 100644
--- a/server/src/journal.rs
+++ b/server/src/journal.rs
@@ -5,7 +5,6 @@ use std::sync::Arc;
use tokio::sync::{mpsc, oneshot};
use crate::{
- entity_store::EntityStore,
event_log::EventLog,
events::{event_timestamp, Event, EventRecord},
projection_apply,
@@ -25,7 +24,6 @@ pub struct JournalClient {
impl JournalClient {
pub fn spawn(
event_log: Arc<EventLog>,
- entity_store: EntityStore,
projection_store: ProjectionStore,
next_seq: u64,
) -> Self {
@@ -33,7 +31,6 @@ impl JournalClient {
tokio::spawn(journal_worker(
rx,
event_log,
- entity_store,
projection_store,
next_seq,
));
@@ -62,7 +59,6 @@ impl JournalClient {
async fn journal_worker(
mut rx: mpsc::Receiver<JournalCommand>,
event_log: Arc<EventLog>,
- entity_store: EntityStore,
projection_store: ProjectionStore,
mut next_seq: u64,
) {
@@ -75,7 +71,6 @@ async fn journal_worker(
let result = append_and_project_batch(
&event_log,
&projection_store,
- &entity_store,
&mut next_seq,
&batch,
)
@@ -99,7 +94,6 @@ async fn journal_worker(
async fn append_and_project_batch(
event_log: &EventLog,
projection_store: &ProjectionStore,
- entity_store: &EntityStore,
next_seq: &mut u64,
commands: &[JournalCommand],
) -> Result<(), String> {
@@ -117,7 +111,7 @@ async fn append_and_project_batch(
.await
.map_err(|e| e.to_string())?;
*next_seq = seq;
- projection_apply::apply_records(projection_store, entity_store, &records)
+ projection_apply::apply_records(projection_store, &records)
.map_err(|e| format!("projection apply failed after durable append: {e}"))
}
@@ -132,10 +126,9 @@ mod tests {
let log_path = tmp.path().join("events.jsonl");
let event_log = Arc::new(EventLog::new(log_path));
let db = durable::Db::open(tmp.path().join("store")).unwrap();
- let entity_store = EntityStore::from_db(&db).unwrap();
let projection_store = ProjectionStore::from_db(&db).unwrap();
- let journal = JournalClient::spawn(event_log, entity_store, projection_store.clone(), 1);
+ let journal = JournalClient::spawn(event_log, projection_store.clone(), 1);
let j1 = journal.clone();
let j2 = journal.clone();
@@ -177,11 +170,9 @@ mod tests {
.unwrap();
let db = durable::Db::open(tmp.path().join("store")).unwrap();
- let entity_store = EntityStore::from_db(&db).unwrap();
let projection_store = ProjectionStore::from_db(&db).unwrap();
projection_apply::apply_records(
&projection_store,
- &entity_store,
&[EventRecord::new(
1,
1,
@@ -196,7 +187,6 @@ mod tests {
let journal = JournalClient::spawn(
event_log.clone(),
- entity_store,
projection_store.clone(),
next_seq,
);
@@ -219,10 +209,9 @@ mod tests {
let log_path = tmp.path().join("events.jsonl");
let event_log = Arc::new(EventLog::new(log_path));
let db = durable::Db::open(tmp.path().join("store")).unwrap();
- let entity_store = EntityStore::from_db(&db).unwrap();
let projection_store = ProjectionStore::from_db(&db).unwrap();
let journal =
- JournalClient::spawn(event_log.clone(), entity_store, projection_store.clone(), 1);
+ JournalClient::spawn(event_log.clone(), projection_store.clone(), 1);
journal
.append_many(vec![
diff --git a/server/src/lib.rs b/server/src/lib.rs
index 5c02c8e704e4664453bad75d819df8a067668176..3dfc7c8acb8ed61bb73ade63e72768e402042cc5 100644
--- a/server/src/lib.rs
+++ b/server/src/lib.rs
@@ -1,5 +1,4 @@
pub mod api;
-pub mod entity_store;
pub mod event_log;
pub mod events;
pub mod fetch;
diff --git a/server/src/projection_apply.rs b/server/src/projection_apply.rs
index 5644557a41b3e9497c7421b444155ae629fa79f1..9c8990a8af927f35d3344c8d0872a516aba56b86 100644
--- a/server/src/projection_apply.rs
+++ b/server/src/projection_apply.rs
@@ -1,22 +1,20 @@
//! Apply event-log records to the durable projection as precise point updates.
//!
//! Each batch of records lowers to reified durable writes (edge merges, child
-//! links, voted-pair flags, recent-vote pushes, entity payloads) plus a cursor
-//! advance, all committed in one atomic `DisableWal` batch. The cursor moving in
-//! the same batch as the (non-idempotent) edge merges guarantees e
… preview truncated; 33,427 characters omittedHardlinks — judgments / attempts / prompt
judgments
attempts
Prompt text is loaded only by the download route.