{"record":{"id":"abfa20aa6000bd4c","repo":"affaan-m/ECC","slug":"atomic-rollback-compare-and-swap-failed","errorCode":null,"errorMessage":"atomic rollback compare-and-swap failed","messagePattern":"atomic rollback compare-and-swap failed","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"critical","filePath":"ecc2/src/session/store.rs","lineNumber":5529,"sourceCode":"                anyhow::bail!(\"health check result does not match persisted assertion\");\n            }\n            Ok(healthy)\n        });\n        let healthy = matches!(health_result, Ok(true));\n        let event_type = match &health_result {\n            Ok(true) => \"promoted\",\n            Ok(false) => \"promotion_rolled_back\",\n            Err(_) => \"health_check_error_rolled_back\",\n        };\n        let health_check_status = match &health_result {\n            Ok(true) => \"healthy\",\n            Ok(false) => \"unhealthy\",\n            Err(_) => \"error\",\n        };\n        if !healthy {\n            let restored = tx.execute(\"UPDATE active_harness_config SET candidate_id = ?1, updated_at = ?2 WHERE slot = 'default' AND candidate_id = ?3\", rusqlite::params![stored_baseline_id, now, stored_candidate_id])?;\n            if restored != 1 {\n                anyhow::bail!(\"atomic rollback compare-and-swap failed\");\n            }\n        }\n        let health_json = health_evidence.canonical_json()?;\n        let health_digest = health_evidence.digest()?;\n        tx.execute(\"INSERT INTO harness_evaluations (candidate_id, baseline_id, evaluator, samples_json, policy_json, comparison_json, evidence_ref, health_evidence_json, health_evidence_sha256, asserted_health, health_check_status, legacy_unverifiable, created_at) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, 0, ?12)\", rusqlite::params![stored_candidate_id, stored_baseline_id, evaluator, serde_json::to_string(samples)?, serde_json::to_string(&policy)?, serde_json::to_string(&comparison)?, evidence_ref, health_json, health_digest, health_evidence.asserted_healthy, health_check_status, now])?;\n        let evaluation_id = tx.last_insert_rowid();\n        tx.execute(\"INSERT INTO harness_eval_audit (event_type, candidate_id, prior_candidate_id, evaluation_id, evidence_ref, health_evidence_json, health_evidence_sha256, asserted_health, health_check_status, legacy_unverifiable, created_at) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, 0, ?10)\", rusqlite::params![event_type, stored_candidate_id, stored_baseline_id, evaluation_id, evidence_ref, health_json, health_digest, health_evidence.asserted_healthy, health_check_status, now])?;\n        tx.commit()?;\n        let failures = match health_result {\n            Ok(true) => Vec::new(),\n            Ok(false) => vec![\"post-promotion health check returned false\".to_string()],\n            Err(error) => vec![format!(\"health check error: {error:#}\")],\n        };\n        Ok(HarnessPromotionOutcome {\n            evaluation_id: Some(evaluation_id),\n            promoted: healthy,\n            rolled_back: !healthy,\n            failures,","sourceCodeStart":5511,"sourceCodeEnd":5547,"githubUrl":"https://github.com/affaan-m/ECC/blob/01e15490f04e29cfefe3896951f43db46994d8ee/ecc2/src/session/store.rs#L5511-L5547","documentation":"When a post-promotion health check fails, the store atomically rolls back: UPDATE active_harness_config SET candidate_id=baseline WHERE slot='default' AND candidate_id=new. If restored != 1, the row was concurrently changed between the promotion CAS and the rollback CAS, so the rollback could not complete and the live config state is now ambiguous.","triggerScenarios":"Between the promotion CAS and the rollback CAS inside promote_harness, another writer changed active_harness_config (concurrent promotion/rollback or manual edit).","commonSituations":"Two concurrent promoters; a manual DB edit during a failed promotion; the active row was deleted mid-flow.","solutions":["Treat this as a critical inconsistency: re-read active_harness_config and reconcile the live state manually before any further promotion.","Serialize all harness mutations behind a single writer so a rollback can never race another mutation.","Alert loudly — the system cannot guarantee which candidate is active after this failure."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// You cannot validate away a concurrent mid-rollback mutation. Best pre-call guard is\n// to guarantee single-writer access to active_harness_config:\nlet _guard = harness_write_lock.lock().unwrap();\nstore.promote_harness(&candidate_id, &baseline_id, ...)?;","typeGuard":null,"tryCatchPattern":"match store.promote_harness(&candidate_id, &baseline_id, evaluator, evidence_ref, &health_evidence, probe) {\n    Ok(out) => { /* handled */ }\n    Err(e) if e.to_string().contains(\"atomic rollback compare-and-swap failed\") => {\n        // CRITICAL: live config state is ambiguous. Stop, re-read active_harness_config,\n        // reconcile manually, and alert — do not auto-retry blindly.\n        let active = read_active_config(&conn)?;\n        return Err(anyhow::anyhow!(\"rollback CAS failed; reconcile active config {active:?}\"));\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Serialize all writes to active_harness_config behind a single writer / distributed lock.","Treat a rollback-CAS failure as a page-worthy incident, not a transient retry.","After any rollback-CAS failure, freeze promotions until an operator reconciles the live state."],"tags":["rust","sqlite","harness","concurrency","cas","rollback"],"backgroundTag":null,"analyzedSha":"01e15490f04e29cfefe3896951f43db46994d8ee","analyzedAt":"2026-08-13T00:31:08.655Z","schemaVersion":2},"datasetVersion":"2026-08-13T04:17:16.726Z"}