JuliusBrussee/caveman · critical

proposalrun: seq %d row_hash mismatch (tampered)

Error message

proposalrun: seq %d row_hash mismatch (tampered)

What it means

The core tamper signal. VerifyChain recomputed the row's hash from its stored fields (PrevHash, Seq, Action, Detail, CostUSD, CreatedAt) and got a value different from the stored RowHash. Since every input is read from the row itself, a mismatch means at least one field was modified after the row was written without updating its hash — the chain's integrity guarantee is broken at this row and everything after it is unverifiable.

Source

Thrown at shared/platform/proposalrun/proposalrun.go:364

	sorted := make([]Run, len(runs))
	copy(sorted, runs)
	sort.Slice(sorted, func(i, j int) bool { return sorted[i].Seq < sorted[j].Seq })

	prevHash := ""
	for i, r := range sorted {
		wantSeq := int64(i + 1)
		if r.Seq != wantSeq {
			return fmt.Errorf("proposalrun: seq %d out of order (expected %d)", r.Seq, wantSeq)
		}
		if r.PrevHash != prevHash {
			return fmt.Errorf("proposalrun: seq %d prev_hash does not link to the prior row", r.Seq)
		}
		got, err := RowHash(r.PrevHash, r.Seq, r.Action, r.Detail, r.CostUSD, r.CreatedAt)
		if err != nil {
			return fmt.Errorf("proposalrun: seq %d recompute: %w", r.Seq, err)
		}
		if got != r.RowHash {
			return fmt.Errorf("proposalrun: seq %d row_hash mismatch (tampered)", r.Seq)
		}
		prevHash = r.RowHash
	}
	return nil
}

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Diff the stored row against a trusted copy (PITR / replica / off-site audit log) to identify which field changed and whether it was accidental or malicious.
2. If accidental and authorized, correct the field AND recompute row_hash, then re-hash every subsequent row — or accept the break and record it in the audit trail.
3. Prevent recurrence: revoke UPDATE on the table from the app role (INSERT/SELECT only) and verify the chain on a schedule so drift is caught at first occurrence.

Example fix

// before
-- operator 'fix'
UPDATE proposal_runs SET cost_usd = 0.42 WHERE seq = 7;  -- row_hash now stale -> mismatch

// after
-- make the table append-only for the app role
REVOKE UPDATE ON proposal_runs FROM app_role;
-- corrections go through the package API, which re-chains rows

// Optional early self-check: recompute a row's hash before trusting it downstream.
func rowSelfConsistent(r proposalrun.Run) bool {
    got, err := proposalrun.RowHash(r.PrevHash, r.Seq, r.Action, r.Detail, r.CostUSD, r.CreatedAt)
    return err == nil && got == r.RowHash
}

if err := proposalrun.VerifyChain(runs); err != nil {
    if strings.Contains(err.Error(), "row_hash mismatch") {
        // treat as security incident: snapshot the row, diff against PITR/replica, alert
    }
}

When it happens

Trigger: Any out-of-band UPDATE to action, detail, cost_usd, or created_at on a proposal_runs row; a DB restore that lost precision (e.g. created_at truncated to milliseconds by a different engine, breaking the microsecond-pinned formatTime); floating-point cost_usd round-tripped through a column with different precision.

Common situations: Manual SQL 'fixes' by an operator; migrating the table through a system that coerces timestamptz or numeric types; a restore from a backup tool that doesn't preserve exact microsecond timestamps; genuine malicious tampering.

Related errors

• proposalrun: seq %d out of order (expected %d)
• proposalrun: seq %d prev_hash does not link to the prior row
• cave_mastra_terminal_failure
• cave_harness_incomplete_evidence
• cave_vercel_usage_missing

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/8a746998cab53161. Report an issue: GitHub.