JuliusBrussee/caveman · critical

proposalrun: seq %d out of order (expected %d)

Error message

proposalrun: seq %d out of order (expected %d)

What it means

VerifyChain re-establishes the hash chain over a slice of Run rows. It sorts by Seq, then requires Seq to be exactly 1,2,3,... with no gaps or duplicates. This error means the sorted sequence is not contiguous from 1: a row is missing (deleted), Seq values start above 1 (rows written after a purge without resetting sequence), or duplicates exist so numbering cannot line up with position.

Source

Thrown at shared/platform/proposalrun/proposalrun.go:354

// VerifyChain re-walks a proposal's runs ordered by seq and asserts:
//   - seq starts at 1 and increases by exactly 1 (no gaps, no dupes);
//   - each row's prev_hash equals the previous row's row_hash ("" for seq 1);
//   - each row's row_hash recomputes from its own stored fields (no tampering).
//
// It returns nil for an empty slice (a proposal with no runs yet is consistent).
func VerifyChain(runs []Run) error {
	if len(runs) == 0 {
		return nil
	}
	sorted := make([]Run, len(runs))
	copy(sorted, runs)
	sort.Slice(sorted, func(i, j int) bool { return sorted[i].Seq < sorted[j].Seq })

	prevHash := ""
	for i, r := range sorted {
		wantSeq := int64(i + 1)
		if r.Seq != wantSeq {
			return fmt.Errorf("proposalrun: seq %d out of order (expected %d)", r.Seq, wantSeq)
		}
		if r.PrevHash != prevHash {
			return fmt.Errorf("proposalrun: seq %d prev_hash does not link to the prior row", r.Seq)
		}
		got, err := RowHash(r.PrevHash, r.Seq, r.Action, r.Detail, r.CostUSD, r.CreatedAt)
		if err != nil {
			return fmt.Errorf("proposalrun: seq %d recompute: %w", r.Seq, err)
		}
		if got != r.RowHash {
			return fmt.Errorf("proposalrun: seq %d row_hash mismatch (tampered)", r.Seq)
		}
		prevHash = r.RowHash
	}
	return nil
}

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Load the runs without WHERE filters — the chain is only defined over the complete ordered set.
2. If rows were legitimately deleted, renumber Seq contiguously from 1 and recompute PrevHash/RowHash for every subsequent row (a full re-chain), or keep gaps and change the verifier contract.
3. Check for duplicate Seq with SELECT seq, count(*) ... GROUP BY seq HAVING count(*) > 1 and fix the writer that produced them.

Example fix

// before
runs, _ := store.ListRuns(ctx, orgID, func(r Run) bool { return r.Action != "draft" }) // filters break contiguity
err := proposalrun.VerifyChain(runs)

// after
runs, err := store.ListRuns(ctx, orgID) // full set, unfiltered
if err != nil { return err }
err = proposalrun.VerifyChain(runs)

func seqsContiguous(runs []proposalrun.Run) bool {
    seen := make(map[int64]bool, len(runs))
    for _, r := range runs {
        if seen[r.Seq] { return false } // duplicate
        seen[r.Seq] = true
    }
    for i := int64(1); i <= int64(len(runs)); i++ {
        if !seen[i] { return false } // gap
    }
    return true
}
// call before VerifyChain: if !seqsContiguous(runs) { /* the query filtered rows */ }

if err := proposalrun.VerifyChain(runs); err != nil {
    if strings.HasPrefix(err.Error(), "proposalrun: seq") && strings.Contains(err.Error(), "out of order") {
        // check for a WHERE filter in the load query or duplicate seq in the table
    }
}

When it happens

Trigger: Call VerifyChain(runs) where runs was loaded with a filter that dropped rows (e.g. WHERE action != 'x'), after a manual DELETE left a gap, after a partial table copy, or after concurrent writers assigned duplicate Seq values.

Common situations: Audit/verification job run against a filtered query instead of the full table; a bug in the INSERT ... SELECT that allocates Seq; restoring a subset of rows from backup; splitting the table across shards and verifying one shard in isolation.

Related errors

• proposalrun: seq %d prev_hash does not link to the prior row
• proposalrun: seq %d row_hash mismatch (tampered)
• cave_budget_release_reason_required
• ${where} hard chain edge ${edge.from}->${edge.to} is not a h
• ${where} is diagnosis-only but carries a change set, eval pa

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/35ce25e5d9a56d3f. Report an issue: GitHub.