hashicorp/nomad · critical

stable key mismatch: source= , destination=

Error message

stable key %s mismatch: source=%d, destination=%d

What it means

verifyMigration compares CurrentTerm and LastVoteTerm between source and destination stable stores after a migration; this error (migrate.go:294) means the uint64 values differ. The migration is aborted because divergent term/vote state can cause Raft voting invariant violations or lost leadership state. The error names the key and both observed values.

Solutions

1. Fully stop the Raft node before migrating so term cannot change mid-copy.
2. Delete or empty the destination store so stale CurrentTerm/LastVoteTerm cannot survive.
3. Check that MigrateToWAL's copy loop persists both uint64 keys before verification.
4. Use the printed source/destination values to determine whether the destination is stale or the copy dropped the key.
5. Re-run the migration and confirm verifyMigration returns nil.

Example fix

// before
raftutil.MigrateToWAL(src, existingDstPath) // dst holds stale CurrentTerm from old node
// after
raft.Shutdown() // halt raft so term cannot change
os.Remove(existingDstPath)   // remove stale destination
raftutil.MigrateToWAL(src, existingDstPath)

// Before migrating, ensure the destination has no stale stable state:
for _, key := range []string{"CurrentTerm", "LastVoteTerm"} {
    if _, err := dst.GetUint64([]byte(key)); err == nil {
        return fmt.Errorf("destination already holds %s; use a fresh store", key)
    }
}

if err := raftutil.MigrateToWAL(src, dstPath); err != nil {
    if strings.Contains(err.Error(), "stable key") && strings.Contains(err.Error(), "mismatch") {
        // destination is stale or copy dropped a key; wipe and retry
        os.Remove(dstPath)
    }
    return err
}

When it happens

Trigger: After MigrateToWAL, dst.GetUint64(key) differs from src.GetUint64(key) for "CurrentTerm" or "LastVoteTerm" — the copy wrote a stale/zero value, the destination retained data from a previous use, or the node kept running and the term advanced between copy and verify.

Common situations: Migrating into a reused bolt DB holding state from an earlier node/cluster; running migration while the Raft node is live; a copy-loop bug skipping uint64 stable keys.

Related errors

• data verification failed
• failed to copy stable store
• failed to get destination log
• failed to get destination uint64 key
• log entry mismatch

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/5c5035259ada2889. Report an issue: GitHub.

Appendix: source

Thrown at helper/raftutil/migrate.go:294

	if srcLast != dstLast {
		return fmt.Errorf("last index mismatch: source=%d, destination=%d", srcLast, dstLast)
	}

	// Verify stable store uint64 keys.
	uint64Keys := []string{"CurrentTerm", "LastVoteTerm"}
	for _, key := range uint64Keys {
		srcVal, err := src.GetUint64([]byte(key))
		if err != nil {
			return fmt.Errorf("failed to get source uint64 key %s: %w", key, err)
		}

		dstVal, err := dst.GetUint64([]byte(key))
		if err != nil {
			return fmt.Errorf("failed to get destination uint64 key %s: %w", key, err)
		}

		if srcVal != dstVal {
			return fmt.Errorf("stable key %s mismatch: source=%d, destination=%d", key, srcVal, dstVal)
		}
	}

	// Verify stable store byte keys.
	byteKeys := []string{"LastVoteCand"}
	for _, key := range byteKeys {
		srcVal, err := src.Get([]byte(key))
		if err != nil {
			srcVal = nil
		}

		dstVal, err := dst.Get([]byte(key))
		if err != nil {
			dstVal = nil
		}

		if string(srcVal) != string(dstVal) {
			return fmt.Errorf("stable key %s mismatch: source=%q, destination=%q", key, srcVal, dstVal)

View on GitHub (pinned to 482b49bf1a)