JuliusBrussee/caveman · critical

proposalrun: seq out of order (expected )

Error message

proposalrun: seq %d out of order (expected %d)

What it means

VerifyChain re-walks a proposal's run rows sorted by seq and requires seq to start at 1 and increase by exactly 1. This error means the stored chain has a gap, a duplicate seq, or a seq that does not start at 1 — i.e. the audit hash chain's sequence integrity is broken (rows were deleted, inserted out of band, or written concurrently without proper sequencing).

Solutions

1. Identify the offending seq from the error and restore the missing/duplicate row from backup or the append API.
2. Re-append missing runs using the library's Append path so seq and prev_hash are recomputed correctly.
3. If rows were deleted intentionally, rebuild the chain from the authoritative event log rather than patching seqs by hand.
4. Add a DB constraint (unique seq per proposal) and use serialized/transactional appends to prevent recurrence.

Example fix

// before
db.Exec("DELETE FROM proposal_runs WHERE id = $1", id) // leaves a seq gap
// after
// soft-delete or append a compensating run so seq 1..N stays contiguous

func seqsContiguous(runs []proposalrun.Run) bool {
	if len(runs) == 0 { return true }
	seen := map[int64]bool{}
	for _, r := range runs {
		if r.Seq < 1 || r.Seq > int64(len(runs)) || seen[r.Seq] { return false }
		seen[r.Seq] = true
	}
	return len(seen) == len(runs)
}

if err := proposalrun.VerifyChain(runs); err != nil {
	if strings.Contains(err.Error(), "out of order") {
		// repair: re-append missing rows or restore from backup
	}
	return fmt.Errorf("chain verify: %w", err)
}

When it happens

Trigger: Calling VerifyChain(runs) where sorted Seqs are not 1..N: a row was hard-deleted, a duplicate seq exists, the first row's seq != 1, or rows from another proposal were mixed in.

Common situations: Manual DB cleanup deleting rows, failed/partial inserts leaving gaps, concurrent writers both claiming the next seq, migration scripts renumbering rows, copying rows between proposals.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• proposalrun: seq prev_hash does not link to the prior row
• proposalrun: seq recompute
• proposalrun: seq row_hash mismatch (tampered)
• cave_harness_incomplete_evidence
• cave_runtime_segment_id_collision

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/35ce25e5d9a56d3f. Report an issue: GitHub.

Appendix: source

Thrown at shared/platform/proposalrun/proposalrun.go:354

// VerifyChain re-walks a proposal's runs ordered by seq and asserts:
//   - seq starts at 1 and increases by exactly 1 (no gaps, no dupes);
//   - each row's prev_hash equals the previous row's row_hash ("" for seq 1);
//   - each row's row_hash recomputes from its own stored fields (no tampering).
//
// It returns nil for an empty slice (a proposal with no runs yet is consistent).
func VerifyChain(runs []Run) error {
	if len(runs) == 0 {
		return nil
	}
	sorted := make([]Run, len(runs))
	copy(sorted, runs)
	sort.Slice(sorted, func(i, j int) bool { return sorted[i].Seq < sorted[j].Seq })

	prevHash := ""
	for i, r := range sorted {
		wantSeq := int64(i + 1)
		if r.Seq != wantSeq {
			return fmt.Errorf("proposalrun: seq %d out of order (expected %d)", r.Seq, wantSeq)
		}
		if r.PrevHash != prevHash {
			return fmt.Errorf("proposalrun: seq %d prev_hash does not link to the prior row", r.Seq)
		}
		got, err := RowHash(r.PrevHash, r.Seq, r.Action, r.Detail, r.CostUSD, r.CreatedAt)
		if err != nil {
			return fmt.Errorf("proposalrun: seq %d recompute: %w", r.Seq, err)
		}
		if got != r.RowHash {
			return fmt.Errorf("proposalrun: seq %d row_hash mismatch (tampered)", r.Seq)
		}
		prevHash = r.RowHash
	}
	return nil
}

View on GitHub (pinned to 3ee70a1026)