affaan-m/ECC · error · anyhow::Error

health check result does not match persisted assertion

Error message

health check result does not match persisted assertion

What it means

After the promotion CAS succeeds, promote_harness invokes the caller-supplied health_check closure and requires its bool result to equal health_evidence.asserted_healthy. The persisted assertion must agree with the live probe taken at promotion time.

Source

Thrown at ecc2/src/session/store.rs:5511

        if !comparison.passed {
            tx.execute("INSERT INTO harness_evaluations (candidate_id, baseline_id, evaluator, samples_json, policy_json, comparison_json, evidence_ref, legacy_unverifiable, created_at) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, 0, ?8)", rusqlite::params![stored_candidate_id, stored_baseline_id, evaluator, serde_json::to_string(samples)?, serde_json::to_string(&policy)?, serde_json::to_string(&comparison)?, evidence_ref, now])?;
            let evaluation_id = tx.last_insert_rowid();
            tx.execute("INSERT INTO harness_eval_audit (event_type, candidate_id, prior_candidate_id, evaluation_id, evidence_ref, legacy_unverifiable, created_at) VALUES ('promotion_rejected', ?1, ?2, ?3, ?4, 0, ?5)", rusqlite::params![stored_candidate_id, stored_baseline_id, evaluation_id, evidence_ref, now])?;
            tx.commit()?;
            return Ok(HarnessPromotionOutcome {
                evaluation_id: Some(evaluation_id),
                promoted: false,
                rolled_back: false,
                failures: comparison.failures,
            });
        }
        let changed = tx.execute("UPDATE active_harness_config SET candidate_id = ?1, updated_at = ?2 WHERE slot = 'default' AND candidate_id = ?3", rusqlite::params![stored_candidate_id, now, stored_baseline_id])?;
        if changed != 1 {
            anyhow::bail!("atomic promotion compare-and-swap failed");
        }
        let health_result = health_check(candidate_id).and_then(|healthy| {
            if healthy != health_evidence.asserted_healthy {
                anyhow::bail!("health check result does not match persisted assertion");
            }
            Ok(healthy)
        });
        let healthy = matches!(health_result, Ok(true));
        let event_type = match &health_result {
            Ok(true) => "promoted",
            Ok(false) => "promotion_rolled_back",
            Err(_) => "health_check_error_rolled_back",
        };
        let health_check_status = match &health_result {
            Ok(true) => "healthy",
            Ok(false) => "unhealthy",
            Err(_) => "error",
        };
        if !healthy {
            let restored = tx.execute("UPDATE active_harness_config SET candidate_id = ?1, updated_at = ?2 WHERE slot = 'default' AND candidate_id = ?3", rusqlite::params![stored_baseline_id, now, stored_candidate_id])?;
            if restored != 1 {
                anyhow::bail!("atomic rollback compare-and-swap failed");

View on GitHub (pinned to 01e15490f0)

Solutions

  1. Re-capture health_evidence immediately before promoting so assertion and live probe observe the same state.
  2. Make the health_check closure deterministic, fast, and free of external flakiness.
  3. If the candidate is genuinely unhealthy, fix the underlying cause before retrying promotion.
Defensive patterns

Strategy: validation

Validate before calling

// Probe health first, then build the assertion from the probe result so they cannot disagree.
let live_healthy = run_health_check(&candidate_id)?;
let health_evidence = HealthEvidenceSnapshot::for_candidate(&candidate_id, evaluator, live_healthy)?;
health_evidence.verify()?;
// now promote — health_check closure will re-run and must agree with live_healthy.

Type guard

fn assertion_matches_probe(asserted: bool, probe: bool) -> bool {
    asserted == probe
}

Try / catch

match store.promote_harness(&candidate_id, &baseline_id, evaluator, evidence_ref, &health_evidence, |id| run_health_check(id)) {
    Ok(out) => { /* handled; out.rolled_back indicates a mismatch-driven rollback */ }
    Err(e) if e.to_string().contains("health check result does not match") => {
        // candidate flapped between capture and probe; re-capture and retry once stable
    }
    Err(e) => return Err(e),
}

Prevention

When it happens

Trigger: health_check(candidate_id) returns a bool different from health_evidence.asserted_healthy (e.g. asserted true but the probe returned false, or vice versa).

Common situations: The candidate became unhealthy between evidence capture and the live check; the health_check closure is flaky or depends on a service that is down; the assertion was optimistically set to true without a real probe.

Related errors

AI-assisted analysis of affaan-m/ECC@01e15490f0 (2026-08-13). Data as JSON: /api/errors/39a2e29c9daecb47. Report an issue: GitHub.