{"record":{"id":"db096a78c0022648","repo":"zeroclaw-labs/zeroclaw","slug":"sequence-gap-at-line-expected-got","errorCode":null,"errorMessage":"sequence gap at line {}: expected {}, got {}","messagePattern":"sequence gap at line (.+?): expected (.+?), got (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/zeroclaw-runtime/src/security/audit.rs","lineNumber":473,"sourceCode":"    let mut expected_prev_hash = GENESIS_PREV_HASH.to_string();\n    let mut expected_sequence: u64 = 0;\n\n    // Attempt to load signing key from environment (optional)\n    let signing_key = std::env::var(\"ZEROCLAW_AUDIT_SIGNING_KEY\")\n        .ok()\n        .and_then(|key_hex| hex::decode(&key_hex).ok())\n        .filter(|key_bytes| key_bytes.len() == 32);\n\n    for (line_idx, line) in reader.lines().enumerate() {\n        let line = line?;\n        if line.trim().is_empty() {\n            continue;\n        }\n        let entry: AuditEvent = serde_json::from_str(&line)?;\n\n        // Check sequence continuity\n        if entry.sequence != expected_sequence {\n            bail!(\n                \"sequence gap at line {}: expected {}, got {}\",\n                line_idx + 1,\n                expected_sequence,\n                entry.sequence\n            );\n        }\n\n        // Check prev_hash linkage\n        if entry.prev_hash != expected_prev_hash {\n            bail!(\n                \"prev_hash mismatch at line {} (sequence {}): expected {}, got {}\",\n                line_idx + 1,\n                entry.sequence,\n                expected_prev_hash,\n                entry.prev_hash\n            );\n        }\n","sourceCodeStart":455,"sourceCodeEnd":491,"githubUrl":"https://github.com/zeroclaw-labs/zeroclaw/blob/88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc/crates/zeroclaw-runtime/src/security/audit.rs#L455-L491","documentation":"verify_chain walked the audit log and found an entry whose sequence number is not the expected next value (previous + 1). The audit log is an append-only chain; a sequence gap means whole lines are missing — deleted entries, a truncated/spliced file, or two writers interleaving — and the chain can no longer be trusted as a complete record.","triggerScenarios":"Someone deleted one or more lines from the JSONL audit log; log rotation/truncation cut the file mid-way; a crash followed by incorrect manual 'repair'; two daemon instances writing the same log without the endpoint lock.","commonSituations":"Post-incident cleanup that edits audit logs; disk-full followed by partial writes and manual pruning; copying/restoring an old version of the log over a newer one; operators testing tamper detection.","solutions":["Restore the log from backup/snapshot that includes the missing sequences.","If tampering is plausible, preserve the file as evidence and alert security rather than editing it.","If the gap is accepted (e.g. documented log loss), archive the broken log and start a fresh chain per your recovery policy.","Ensure exactly one daemon writes the log (endpoint lock) and that rotation never truncates mid-file."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if let Err(e) = audit.verify_chain() {\n    // stop appending, preserve evidence, quarantine, alert — never auto-repair\n    tracing::error!(\"audit chain verification failed: {e}\");\n    std::fs::rename(&log_path, log_path.with_extension(\"quarantine\"))?;\n    alert_security(&e.to_string());\n}","preventionTips":["Run verify_chain on a schedule and at startup; treat any failure as an incident.","Keep exactly one writer per audit log (enforced by the endpoint lock).","Use append-only/immutable storage (WORM, chattr +a, or a log service) for audit files.","Take regular snapshots so gaps can be diffed and recovered."],"tags":["audit","integrity","hash-chain","tamper-detection","verification"],"backgroundTag":"sequence-number-gap","analyzedSha":"88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc","analyzedAt":"2026-08-23T01:07:41.857Z","schemaVersion":2},"datasetVersion":"2026-08-23T08:06:27.607Z"}