hashicorp/nomad · critical

stable key %s mismatch: source=%q, destination=%q

Error message

stable key %s mismatch: source=%q, destination=%q

What it means

verifyMigration compares the byte-valued stable key "LastVoteCand" between source and destination with %q formatting (migrate.go:312); this error fires when the candidate IDs differ, including nil vs non-nil when one store errors on Get (errors are treated as nil on both sides). The migration aborts because divergent last-vote candidate state can violate Raft voting invariants.

Source

Thrown at helper/raftutil/migrate.go:312

			return fmt.Errorf("stable key %s mismatch: source=%d, destination=%d", key, srcVal, dstVal)
		}
	}

	// Verify stable store byte keys.
	byteKeys := []string{"LastVoteCand"}
	for _, key := range byteKeys {
		srcVal, err := src.Get([]byte(key))
		if err != nil {
			srcVal = nil
		}

		dstVal, err := dst.Get([]byte(key))
		if err != nil {
			dstVal = nil
		}

		if string(srcVal) != string(dstVal) {
			return fmt.Errorf("stable key %s mismatch: source=%q, destination=%q", key, srcVal, dstVal)
		}
	}

	// If we have logs, spot-check a few entries for data integrity.
	if srcFirst > 0 && srcLast > 0 {
		indicesToCheck := []uint64{srcFirst}
		if srcLast > srcFirst {
			// Check middle entry.
			middle := srcFirst + (srcLast-srcFirst)/2
			indicesToCheck = append(indicesToCheck, middle)
			// Check last entry.
			indicesToCheck = append(indicesToCheck, srcLast)
		}

		for _, idx := range indicesToCheck {
			var srcLog, dstLog raft.Log
			if err := src.GetLog(idx, &srcLog); err != nil {
				return fmt.Errorf("failed to get source log %d: %w", idx, err)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Ensure the destination store is empty/fresh so no stale LastVoteCand survives.
2. Check that MigrateToWAL copies the LastVoteCand byte key and preserves absence on both sides.
3. Match the destination store's Get error semantics to the source's for missing keys.
4. Stop the Raft node first so no vote is cast between copy and verification.
5. Inspect the %q values in the error to see if the destination is stale or the copy truncated the value.

Example fix

// before
dst, _ := raftboltdb.NewBoltStore(oldPath) // contains LastVoteCand from previous run
raftutil.MigrateToWAL(src, dst)
// after
deleteStaleStableKeys(dst, []string{"LastVoteCand"}) // or start with a fresh bolt file
raftutil.MigrateToWAL(src, dst)

// Detect divergent LastVoteCand semantics before migrating:
srcCand, srcErr := src.Get([]byte("LastVoteCand"))
dstCand, dstErr := dst.Get([]byte("LastVoteCand"))
if (srcErr == nil) != (dstErr == nil) || string(srcCand) != string(dstCand) {
    return fmt.Errorf("LastVoteCand diverges before migration; clean the destination")
}

if err := raftutil.MigrateToWAL(src, dstPath); err != nil {
    if strings.Contains(err.Error(), "LastVoteCand mismatch") {
        log.Printf("last-vote state diverged: %v", err)
        // wipe destination and retry with the node stopped
    }
    return err
}

When it happens

Trigger: src.Get([]byte("LastVoteCand")) and dst.Get([]byte("LastVoteCand")) return different byte slices after copy — the copy dropped or truncated the key, the destination holds an old value, or one store errored (treated as nil) while the other held a candidate ID.

Common situations: Migrating into a non-empty destination store; a destination whose Get error semantics differ for missing keys (ErrKeyNotFound vs nil) making an unset source key appear as a mismatch; partial copy where byte keys were skipped.

Related errors

• failed to copy stable store: %w
• stable key %s mismatch: source=%d, destination=%d
• failed to open BoltDB store: %w
• failed to copy logs: %w
• data verification failed: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8d984bd039ce3366. Report an issue: GitHub.