affaan-m/ECC · critical · anyhow::Error

atomic rollback compare-and-swap failed

Error message

atomic rollback compare-and-swap failed

What it means

When a post-promotion health check fails, the store atomically rolls back: UPDATE active_harness_config SET candidate_id=baseline WHERE slot='default' AND candidate_id=new. If restored != 1, the row was concurrently changed between the promotion CAS and the rollback CAS, so the rollback could not complete and the live config state is now ambiguous.

Source

Thrown at ecc2/src/session/store.rs:5529

                anyhow::bail!("health check result does not match persisted assertion");
            }
            Ok(healthy)
        });
        let healthy = matches!(health_result, Ok(true));
        let event_type = match &health_result {
            Ok(true) => "promoted",
            Ok(false) => "promotion_rolled_back",
            Err(_) => "health_check_error_rolled_back",
        };
        let health_check_status = match &health_result {
            Ok(true) => "healthy",
            Ok(false) => "unhealthy",
            Err(_) => "error",
        };
        if !healthy {
            let restored = tx.execute("UPDATE active_harness_config SET candidate_id = ?1, updated_at = ?2 WHERE slot = 'default' AND candidate_id = ?3", rusqlite::params![stored_baseline_id, now, stored_candidate_id])?;
            if restored != 1 {
                anyhow::bail!("atomic rollback compare-and-swap failed");
            }
        }
        let health_json = health_evidence.canonical_json()?;
        let health_digest = health_evidence.digest()?;
        tx.execute("INSERT INTO harness_evaluations (candidate_id, baseline_id, evaluator, samples_json, policy_json, comparison_json, evidence_ref, health_evidence_json, health_evidence_sha256, asserted_health, health_check_status, legacy_unverifiable, created_at) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, 0, ?12)", rusqlite::params![stored_candidate_id, stored_baseline_id, evaluator, serde_json::to_string(samples)?, serde_json::to_string(&policy)?, serde_json::to_string(&comparison)?, evidence_ref, health_json, health_digest, health_evidence.asserted_healthy, health_check_status, now])?;
        let evaluation_id = tx.last_insert_rowid();
        tx.execute("INSERT INTO harness_eval_audit (event_type, candidate_id, prior_candidate_id, evaluation_id, evidence_ref, health_evidence_json, health_evidence_sha256, asserted_health, health_check_status, legacy_unverifiable, created_at) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, 0, ?10)", rusqlite::params![event_type, stored_candidate_id, stored_baseline_id, evaluation_id, evidence_ref, health_json, health_digest, health_evidence.asserted_healthy, health_check_status, now])?;
        tx.commit()?;
        let failures = match health_result {
            Ok(true) => Vec::new(),
            Ok(false) => vec!["post-promotion health check returned false".to_string()],
            Err(error) => vec![format!("health check error: {error:#}")],
        };
        Ok(HarnessPromotionOutcome {
            evaluation_id: Some(evaluation_id),
            promoted: healthy,
            rolled_back: !healthy,
            failures,

View on GitHub (pinned to 01e15490f0)

Solutions

1. Treat this as a critical inconsistency: re-read active_harness_config and reconcile the live state manually before any further promotion.
2. Serialize all harness mutations behind a single writer so a rollback can never race another mutation.
3. Alert loudly — the system cannot guarantee which candidate is active after this failure.

// You cannot validate away a concurrent mid-rollback mutation. Best pre-call guard is
// to guarantee single-writer access to active_harness_config:
let _guard = harness_write_lock.lock().unwrap();
store.promote_harness(&candidate_id, &baseline_id, ...)?;

match store.promote_harness(&candidate_id, &baseline_id, evaluator, evidence_ref, &health_evidence, probe) {
    Ok(out) => { /* handled */ }
    Err(e) if e.to_string().contains("atomic rollback compare-and-swap failed") => {
        // CRITICAL: live config state is ambiguous. Stop, re-read active_harness_config,
        // reconcile manually, and alert — do not auto-retry blindly.
        let active = read_active_config(&conn)?;
        return Err(anyhow::anyhow!("rollback CAS failed; reconcile active config {active:?}"));
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Between the promotion CAS and the rollback CAS inside promote_harness, another writer changed active_harness_config (concurrent promotion/rollback or manual edit).

Common situations: Two concurrent promoters; a manual DB edit during a failed promotion; the active row was deleted mid-flow.

Related errors

• atomic promotion compare-and-swap failed
• baseline is not the active harness configuration
• candidate alias collision with physical candidate id
• candidate alias collision with different immutable target
• legacy candidate id collision with different immutable conte

AI-assisted analysis of affaan-m/ECC@01e15490f0 (2026-08-13). Data as JSON: /api/errors/abfa20aa6000bd4c. Report an issue: GitHub.