What causes "atomic promotion compare-and-swap failed" in affaan-m/ECC?

Retry the whole promotion after re-reading the active baseline (optimistic concurrency).

affaan-m/ECC · error · anyhow::Error

atomic promotion compare-and-swap failed

Error message

atomic promotion compare-and-swap failed

What it means

Promotion uses an atomic compare-and-swap: UPDATE active_harness_config SET candidate_id=new WHERE slot='default' AND candidate_id=baseline. If exactly one row is not updated (changed != 1), the baseline changed between the read and the write — a concurrent mutation won the race.

Source

Thrown at ecc2/src/session/store.rs:5507

        if active != stored_baseline_id {
            anyhow::bail!("baseline is not the active harness configuration");
        }
        let now = chrono::Utc::now().to_rfc3339();
        if !comparison.passed {
            tx.execute("INSERT INTO harness_evaluations (candidate_id, baseline_id, evaluator, samples_json, policy_json, comparison_json, evidence_ref, legacy_unverifiable, created_at) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, 0, ?8)", rusqlite::params![stored_candidate_id, stored_baseline_id, evaluator, serde_json::to_string(samples)?, serde_json::to_string(&policy)?, serde_json::to_string(&comparison)?, evidence_ref, now])?;
            let evaluation_id = tx.last_insert_rowid();
            tx.execute("INSERT INTO harness_eval_audit (event_type, candidate_id, prior_candidate_id, evaluation_id, evidence_ref, legacy_unverifiable, created_at) VALUES ('promotion_rejected', ?1, ?2, ?3, ?4, 0, ?5)", rusqlite::params![stored_candidate_id, stored_baseline_id, evaluation_id, evidence_ref, now])?;
            tx.commit()?;
            return Ok(HarnessPromotionOutcome {
                evaluation_id: Some(evaluation_id),
                promoted: false,
                rolled_back: false,
                failures: comparison.failures,
            });
        }
        let changed = tx.execute("UPDATE active_harness_config SET candidate_id = ?1, updated_at = ?2 WHERE slot = 'default' AND candidate_id = ?3", rusqlite::params![stored_candidate_id, now, stored_baseline_id])?;
        if changed != 1 {
            anyhow::bail!("atomic promotion compare-and-swap failed");
        }
        let health_result = health_check(candidate_id).and_then(|healthy| {
            if healthy != health_evidence.asserted_healthy {
                anyhow::bail!("health check result does not match persisted assertion");
            }
            Ok(healthy)
        });
        let healthy = matches!(health_result, Ok(true));
        let event_type = match &health_result {
            Ok(true) => "promoted",
            Ok(false) => "promotion_rolled_back",
            Err(_) => "health_check_error_rolled_back",
        };
        let health_check_status = match &health_result {
            Ok(true) => "healthy",
            Ok(false) => "unhealthy",
            Err(_) => "error",
        };

View on GitHub (pinned to 01e15490f0)

Solutions

Retry the whole promotion after re-reading the active baseline (optimistic concurrency).
Serialize promotions with an external lock/mutex so only one writer mutates active_harness_config at a time.
Surface a distinct 'concurrent modification, please retry' error to the caller rather than a generic failure.

Defensive patterns

Strategy: retry

Validate before calling

// CAS failures are inherently racy; validate by re-reading immediately before retry.
fn try_promote_with_retry(store: &Store, candidate_id: &str, max_attempts: u8) -> Result<HarnessPromotionOutcome> {
    for _ in 0..max_attempts {
        let baseline = store.active_harness_id()?.ok_or_else(|| anyhow::anyhow!("no active"))?;
        match store.promote_harness(candidate_id, &baseline, /* ... */) {
            Ok(out) => return Ok(out),
            Err(e) if e.to_string().contains("atomic promotion compare-and-swap failed") => continue,
            Err(e) => return Err(e),
        }
    }
    anyhow::bail!("promotion lost the CAS race {max_attempts} times");
}

Try / catch

loop {
    let baseline = store.active_harness_id()?.ok_or_else(|| anyhow::anyhow!("no active"))?;
    match store.promote_harness(&candidate_id, &baseline, ...) {
        Ok(out) => break Ok(out),
        Err(e) if e.to_string().contains("atomic promotion compare-and-swap failed") => continue,
        Err(e) => break Err(e),
    }
}

Prevention

Serialize harness mutations with an external mutex so the CAS rarely races.
Bound the retry count and surface a clear 'concurrent modification' error when exhausted.
Re-read the active baseline on every attempt; never assume it is stable across attempts.

When it happens

Trigger: Between promote_harness's baseline check and its CAS update, another writer changed active_harness_config.candidate_id for slot 'default' (concurrent promotion or rollback).

Common situations: Two promoters racing on the same store; a rollback interleaving between read and CAS; the active row was deleted by a reset.

Related errors

AI-assisted analysis of affaan-m/ECC@01e15490f0 (2026-08-13). Data as JSON: /api/errors/d3b65828725ada7c. Report an issue: GitHub.