{"record":{"id":"39a2e29c9daecb47","repo":"affaan-m/ECC","slug":"health-check-result-does-not-match-persisted-asser","errorCode":null,"errorMessage":"health check result does not match persisted assertion","messagePattern":"health check result does not match persisted assertion","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"ecc2/src/session/store.rs","lineNumber":5511,"sourceCode":"        if !comparison.passed {\n            tx.execute(\"INSERT INTO harness_evaluations (candidate_id, baseline_id, evaluator, samples_json, policy_json, comparison_json, evidence_ref, legacy_unverifiable, created_at) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, 0, ?8)\", rusqlite::params![stored_candidate_id, stored_baseline_id, evaluator, serde_json::to_string(samples)?, serde_json::to_string(&policy)?, serde_json::to_string(&comparison)?, evidence_ref, now])?;\n            let evaluation_id = tx.last_insert_rowid();\n            tx.execute(\"INSERT INTO harness_eval_audit (event_type, candidate_id, prior_candidate_id, evaluation_id, evidence_ref, legacy_unverifiable, created_at) VALUES ('promotion_rejected', ?1, ?2, ?3, ?4, 0, ?5)\", rusqlite::params![stored_candidate_id, stored_baseline_id, evaluation_id, evidence_ref, now])?;\n            tx.commit()?;\n            return Ok(HarnessPromotionOutcome {\n                evaluation_id: Some(evaluation_id),\n                promoted: false,\n                rolled_back: false,\n                failures: comparison.failures,\n            });\n        }\n        let changed = tx.execute(\"UPDATE active_harness_config SET candidate_id = ?1, updated_at = ?2 WHERE slot = 'default' AND candidate_id = ?3\", rusqlite::params![stored_candidate_id, now, stored_baseline_id])?;\n        if changed != 1 {\n            anyhow::bail!(\"atomic promotion compare-and-swap failed\");\n        }\n        let health_result = health_check(candidate_id).and_then(|healthy| {\n            if healthy != health_evidence.asserted_healthy {\n                anyhow::bail!(\"health check result does not match persisted assertion\");\n            }\n            Ok(healthy)\n        });\n        let healthy = matches!(health_result, Ok(true));\n        let event_type = match &health_result {\n            Ok(true) => \"promoted\",\n            Ok(false) => \"promotion_rolled_back\",\n            Err(_) => \"health_check_error_rolled_back\",\n        };\n        let health_check_status = match &health_result {\n            Ok(true) => \"healthy\",\n            Ok(false) => \"unhealthy\",\n            Err(_) => \"error\",\n        };\n        if !healthy {\n            let restored = tx.execute(\"UPDATE active_harness_config SET candidate_id = ?1, updated_at = ?2 WHERE slot = 'default' AND candidate_id = ?3\", rusqlite::params![stored_baseline_id, now, stored_candidate_id])?;\n            if restored != 1 {\n                anyhow::bail!(\"atomic rollback compare-and-swap failed\");","sourceCodeStart":5493,"sourceCodeEnd":5529,"githubUrl":"https://github.com/affaan-m/ECC/blob/8321021c54d670126ce3b2969d5deb880b4b0c2a/ecc2/src/session/store.rs#L5493-L5529","documentation":"After the promotion update succeeds, the store runs the caller-supplied health_check and compares its boolean result with the asserted_healthy value recorded in the pre-verified health evidence snapshot. If the live check disagrees with the persisted assertion (or the check errors in a way that changes the outcome handling), the promotion is treated as unhealthy and rolled back — this error surfaces the assertion/result disagreement.","triggerScenarios":"Calling evaluate_promote_and_health_check with asserted_healthy=true in the snapshot while the supplied health_check closure returns false (or vice versa), or the health_check closure itself returns Err which then conflicts with the recorded assertion during result reconciliation.","commonSituations":"Health evidence captured earlier (asserting healthy) but the service degraded by promotion time; a health_check closure hitting a different environment/endpoint than the one the snapshot was generated against; flaky health probes returning a different answer than at evidence-creation time.","solutions":["Regenerate the health evidence snapshot at promotion time so asserted_healthy matches the current health check result.","Make the health_check closure probe the same target/endpoint/environment the snapshot was asserted against.","Investigate why the candidate is unhealthy (check logs of the health probe); the rollback is intentional — fix the candidate before re-promoting.","If the check is flaky, stabilize it (retries with backoff inside the closure) so results are deterministic at promotion time."],"exampleFix":"// before: stale assertion\nlet snapshot = build_snapshot(cand, \"recorded-v1\", /*asserted_healthy*/ true)?;\nstore.evaluate_promote_and_health_check(&cand, &base, \"recorded-v1\", ..., &snapshot, |id| Ok(probe(id)))?;\n\n// after: assert from a fresh probe so they agree\nlet healthy_now = probe(&cand)?;\nlet snapshot = build_snapshot(&cand, \"recorded-v1\", healthy_now)?;\nstore.evaluate_promote_and_health_check(&cand, &base, \"recorded-v1\", ..., &snapshot, |id| Ok(probe(id)))?;","handlingStrategy":"try-catch","validationCode":"// Assert health from a probe taken immediately before the call so the\n// snapshot's asserted_healthy matches what health_check will return\nlet healthy_now = run_health_probe(&candidate_id)?;\nlet snapshot = HealthEvidenceSnapshot::build(&candidate_id, \"recorded-v1\", healthy_now)?;\nsnapshot.verify()?;","typeGuard":null,"tryCatchPattern":"match outcome {\n    Ok(o) if o.rolled_back => {\n        // promotion applied but rolled back due to assertion mismatch:\n        // inspect health_check output, fix the candidate, rebuild snapshot, retry\n    }\n    Err(e) if e.to_string().contains(\"health check result does not match\") => {\n        // rebuild snapshot from a fresh probe and re-attempt once\n    }\n    other => other?,\n}","preventionTips":["Generate health evidence at promotion time, not ahead of time","Point health_check and the evidence snapshot at the same environment/endpoint","Stabilize flaky probes (internal retries/backoff) so results are deterministic","Treat rollback as the intended safety behavior — investigate candidate health before re-promoting"],"tags":["health-check","evidence","mismatch","rust"],"backgroundTag":"unexpected-response-shape","analyzedSha":"8321021c54d670126ce3b2969d5deb880b4b0c2a","analyzedAt":"2026-09-16T10:08:13.343Z","contentChangedAt":"2026-09-16T10:08:13.343Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}