{"record":{"id":"d3b65828725ada7c","repo":"affaan-m/ECC","slug":"atomic-promotion-compare-and-swap-failed","errorCode":null,"errorMessage":"atomic promotion compare-and-swap failed","messagePattern":"atomic promotion compare-and-swap failed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"ecc2/src/session/store.rs","lineNumber":5507,"sourceCode":"        if active != stored_baseline_id {\n            anyhow::bail!(\"baseline is not the active harness configuration\");\n        }\n        let now = chrono::Utc::now().to_rfc3339();\n        if !comparison.passed {\n            tx.execute(\"INSERT INTO harness_evaluations (candidate_id, baseline_id, evaluator, samples_json, policy_json, comparison_json, evidence_ref, legacy_unverifiable, created_at) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, 0, ?8)\", rusqlite::params![stored_candidate_id, stored_baseline_id, evaluator, serde_json::to_string(samples)?, serde_json::to_string(&policy)?, serde_json::to_string(&comparison)?, evidence_ref, now])?;\n            let evaluation_id = tx.last_insert_rowid();\n            tx.execute(\"INSERT INTO harness_eval_audit (event_type, candidate_id, prior_candidate_id, evaluation_id, evidence_ref, legacy_unverifiable, created_at) VALUES ('promotion_rejected', ?1, ?2, ?3, ?4, 0, ?5)\", rusqlite::params![stored_candidate_id, stored_baseline_id, evaluation_id, evidence_ref, now])?;\n            tx.commit()?;\n            return Ok(HarnessPromotionOutcome {\n                evaluation_id: Some(evaluation_id),\n                promoted: false,\n                rolled_back: false,\n                failures: comparison.failures,\n            });\n        }\n        let changed = tx.execute(\"UPDATE active_harness_config SET candidate_id = ?1, updated_at = ?2 WHERE slot = 'default' AND candidate_id = ?3\", rusqlite::params![stored_candidate_id, now, stored_baseline_id])?;\n        if changed != 1 {\n            anyhow::bail!(\"atomic promotion compare-and-swap failed\");\n        }\n        let health_result = health_check(candidate_id).and_then(|healthy| {\n            if healthy != health_evidence.asserted_healthy {\n                anyhow::bail!(\"health check result does not match persisted assertion\");\n            }\n            Ok(healthy)\n        });\n        let healthy = matches!(health_result, Ok(true));\n        let event_type = match &health_result {\n            Ok(true) => \"promoted\",\n            Ok(false) => \"promotion_rolled_back\",\n            Err(_) => \"health_check_error_rolled_back\",\n        };\n        let health_check_status = match &health_result {\n            Ok(true) => \"healthy\",\n            Ok(false) => \"unhealthy\",\n            Err(_) => \"error\",\n        };","sourceCodeStart":5489,"sourceCodeEnd":5525,"githubUrl":"https://github.com/affaan-m/ECC/blob/8321021c54d670126ce3b2969d5deb880b4b0c2a/ecc2/src/session/store.rs#L5489-L5525","documentation":"Promotion uses an optimistic compare-and-swap: a single UPDATE of active_harness_config requires the slot to still hold the baseline candidate (WHERE candidate_id = baseline) and must affect exactly one row. If the row count is not 1, the active configuration changed between the earlier read and the update (a concurrent promotion/rollback), so the promotion aborts instead of overwriting an unknown state.","triggerScenarios":"Two concurrent evaluate_promote_and_health_check calls (or a concurrent rollback) mutating active_harness_config between the transaction's baseline read and the UPDATE; a baseline that was replaced after the row was read; any external writer updating the slot within the transaction window.","commonSituations":"Multiple CI jobs or operators promoting candidates in parallel against the same store; a health-check-triggered rollback landing between your read and update; scripted re-runs racing each other.","solutions":["Simply retry the whole evaluate_promote_and_health_check call: re-read the current active id and use it as the new baseline.","Coordinate promotions (single writer, lock, or queue) so only one process mutates active_harness_config at a time.","Inspect harness_eval_audit to see which event changed the active slot and reconcile before retrying.","Keep transactions short and avoid long-running work (e.g. slow health checks) outside the compare-and-swap window where possible."],"exampleFix":"// before: fire-and-forget promotion that can race\nstore.evaluate_promote_and_health_check(&cand, &base, ...)?;\n\n// after: retry on CAS failure\nmatch store.evaluate_promote_and_health_check(&cand, &base, ...) {\n    Err(e) if e.to_string().contains(\"compare-and-swap failed\") => {\n        let current = store.active_harness_id()?.unwrap();\n        store.evaluate_promote_and_health_check(&cand, &current, ...)?;\n    }\n    r => r?,\n}","handlingStrategy":"retry","validationCode":"// Preflight: confirm the slot still holds your expected baseline\nlet active: Option<String> = /* query active_harness_config WHERE slot = 'default' */;\nanyhow::ensure!(active.as_deref() == Some(stored_baseline_id), \"active config changed; re-base before promoting\");","typeGuard":null,"tryCatchPattern":"const MAX_RETRIES: usize = 3;\nfor attempt in 0..MAX_RETRIES {\n    match store.evaluate_promote_and_health_check(...) {\n        Err(e) if e.to_string().contains(\"atomic promotion compare-and-swap failed\") && attempt + 1 < MAX_RETRIES => continue,\n        other => { other?; break }\n    }\n}","preventionTips":["Single-writer discipline for active_harness_config (lock or queue promotions)","Keep promotion transactions short; do slow work outside the CAS window","On CAS failure, always re-read state — never assume the update landed"],"tags":["concurrency","optimistic-locking","database","rust"],"backgroundTag":"invalid-state-transition","analyzedSha":"8321021c54d670126ce3b2969d5deb880b4b0c2a","analyzedAt":"2026-09-16T10:08:13.343Z","contentChangedAt":"2026-09-16T10:08:13.343Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}