What causes "sequence gap at line {}: expected {}, got {}" in zeroclaw-labs/zeroclaw?

Restore the log from backup/snapshot that includes the missing sequences.

sequence gap at line {}: expected {}, got {}

Error message

sequence gap at line {}: expected {}, got {}

What it means

verify_chain walked the audit log and found an entry whose sequence number is not the expected next value (previous + 1). The audit log is an append-only chain; a sequence gap means whole lines are missing — deleted entries, a truncated/spliced file, or two writers interleaving — and the chain can no longer be trusted as a complete record.

Source

Thrown at crates/zeroclaw-runtime/src/security/audit.rs:473

    let mut expected_prev_hash = GENESIS_PREV_HASH.to_string();
    let mut expected_sequence: u64 = 0;

    // Attempt to load signing key from environment (optional)
    let signing_key = std::env::var("ZEROCLAW_AUDIT_SIGNING_KEY")
        .ok()
        .and_then(|key_hex| hex::decode(&key_hex).ok())
        .filter(|key_bytes| key_bytes.len() == 32);

    for (line_idx, line) in reader.lines().enumerate() {
        let line = line?;
        if line.trim().is_empty() {
            continue;
        }
        let entry: AuditEvent = serde_json::from_str(&line)?;

        // Check sequence continuity
        if entry.sequence != expected_sequence {
            bail!(
                "sequence gap at line {}: expected {}, got {}",
                line_idx + 1,
                expected_sequence,
                entry.sequence
            );
        }

        // Check prev_hash linkage
        if entry.prev_hash != expected_prev_hash {
            bail!(
                "prev_hash mismatch at line {} (sequence {}): expected {}, got {}",
                line_idx + 1,
                entry.sequence,
                expected_prev_hash,
                entry.prev_hash
            );
        }

View on GitHub (pinned to 88bb9c8533)

Solutions

Restore the log from backup/snapshot that includes the missing sequences.
If tampering is plausible, preserve the file as evidence and alert security rather than editing it.
If the gap is accepted (e.g. documented log loss), archive the broken log and start a fresh chain per your recovery policy.
Ensure exactly one daemon writes the log (endpoint lock) and that rotation never truncates mid-file.

Defensive patterns

Strategy: try-catch

Try / catch

if let Err(e) = audit.verify_chain() {
    // stop appending, preserve evidence, quarantine, alert — never auto-repair
    tracing::error!("audit chain verification failed: {e}");
    std::fs::rename(&log_path, log_path.with_extension("quarantine"))?;
    alert_security(&e.to_string());
}

Prevention

Run verify_chain on a schedule and at startup; treat any failure as an incident.
Keep exactly one writer per audit log (enforced by the endpoint lock).
Use append-only/immutable storage (WORM, chattr +a, or a log service) for audit files.
Take regular snapshots so gaps can be diffed and recovered.

When it happens

Trigger: Someone deleted one or more lines from the JSONL audit log; log rotation/truncation cut the file mid-way; a crash followed by incorrect manual 'repair'; two daemon instances writing the same log without the endpoint lock.

Common situations: Post-incident cleanup that edits audit logs; disk-full followed by partial writes and manual pruning; copying/restoring an old version of the log over a newer one; operators testing tamper detection.

Related errors

AI-assisted analysis of zeroclaw-labs/zeroclaw@88bb9c8533 (2026-08-23). Data as JSON: /api/errors/db096a78c0022648. Report an issue: GitHub.