JuliusBrussee/caveman · critical

proposalrun: seq row_hash mismatch (tampered)

Error message

proposalrun: seq %d row_hash mismatch (tampered)

What it means

VerifyChain recomputes each row's hash with RowHash and compares it to the stored RowHash. A mismatch means one or more of the hashed fields (prev_hash, seq, action, detail, cost_usd, created_at) was modified after the row was written — i.e. the audit record was tampered with or corrupted, or the hash was computed with a different canonicalization than the verifier uses.

Solutions

1. Diff the reported row's fields against the original source/event to find which field changed; restore it from backup or event log.
2. Check for precision changes: created_at must be UTC microsecond-truncated RFC3339Nano and cost_usd fixed-6 as RowHash expects.
3. Verify all writers use the same library version of RowHash (canonicalization changes break old rows).
4. If tampering is suspected, preserve the row for forensics and escalate; do not 'fix' the hash to match the edited data.
5. Lock the table against direct updates (revoke UPDATE, triggers) so fields can't change post-write.

Example fix

// before
createdAt := time.Now() // sub-microsecond, local zone stored → hash mismatch on verify
// after
createdAt := time.Now().UTC().Truncate(time.Microsecond) // matches RowHash's formatTime

ok, _ := func() (bool, error) {
	got, err := proposalrun.RowHash(r.PrevHash, r.Seq, r.Action, r.Detail, r.CostUSD, r.CreatedAt)
	return err == nil && got == r.RowHash, err
}()

if err := proposalrun.VerifyChain(runs); err != nil {
	if strings.Contains(err.Error(), "row_hash mismatch") {
		// preserve row for forensics; do not overwrite hash
	}
	return err
}

When it happens

Trigger: Calling VerifyChain(runs) where RowHash(recomputed) != stored r.RowHash for some seq: any field edited in the DB, created_at rewritten with different precision/timezone, cost_usd stored with lossy rounding, or a mixed library version hashing differently.

Common situations: Manual DB edits to detail or cost, migrations changing timestamptz precision or float storage, timezone handling that shifts created_at, application bug writing cost with rounding before hashing, actual tampering being investigated.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• proposalrun: seq prev_hash does not link to the prior row
• cave_sandbox_tool_definition_mismatch
• integration backup hash mismatch
• proposalrun: seq out of order (expected )
• proposalrun: seq recompute

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/8a746998cab53161. Report an issue: GitHub.

Appendix: source

Thrown at shared/platform/proposalrun/proposalrun.go:364

	sorted := make([]Run, len(runs))
	copy(sorted, runs)
	sort.Slice(sorted, func(i, j int) bool { return sorted[i].Seq < sorted[j].Seq })

	prevHash := ""
	for i, r := range sorted {
		wantSeq := int64(i + 1)
		if r.Seq != wantSeq {
			return fmt.Errorf("proposalrun: seq %d out of order (expected %d)", r.Seq, wantSeq)
		}
		if r.PrevHash != prevHash {
			return fmt.Errorf("proposalrun: seq %d prev_hash does not link to the prior row", r.Seq)
		}
		got, err := RowHash(r.PrevHash, r.Seq, r.Action, r.Detail, r.CostUSD, r.CreatedAt)
		if err != nil {
			return fmt.Errorf("proposalrun: seq %d recompute: %w", r.Seq, err)
		}
		if got != r.RowHash {
			return fmt.Errorf("proposalrun: seq %d row_hash mismatch (tampered)", r.Seq)
		}
		prevHash = r.RowHash
	}
	return nil
}

View on GitHub (pinned to 3ee70a1026)