hashicorp/nomad · critical

stable key %s mismatch: source=%d, destination=%d

Error message

stable key %s mismatch: source=%d, destination=%d

What it means

verifyMigration compares CurrentTerm and LastVoteTerm between source and destination stable stores after a migration; this error (migrate.go:294) means the uint64 values differ. The migration is aborted because divergent term/vote state can cause Raft voting invariant violations or lost leadership state. The error names the key and both observed values.

Source

Thrown at helper/raftutil/migrate.go:294

	if srcLast != dstLast {
		return fmt.Errorf("last index mismatch: source=%d, destination=%d", srcLast, dstLast)
	}

	// Verify stable store uint64 keys.
	uint64Keys := []string{"CurrentTerm", "LastVoteTerm"}
	for _, key := range uint64Keys {
		srcVal, err := src.GetUint64([]byte(key))
		if err != nil {
			return fmt.Errorf("failed to get source uint64 key %s: %w", key, err)
		}

		dstVal, err := dst.GetUint64([]byte(key))
		if err != nil {
			return fmt.Errorf("failed to get destination uint64 key %s: %w", key, err)
		}

		if srcVal != dstVal {
			return fmt.Errorf("stable key %s mismatch: source=%d, destination=%d", key, srcVal, dstVal)
		}
	}

	// Verify stable store byte keys.
	byteKeys := []string{"LastVoteCand"}
	for _, key := range byteKeys {
		srcVal, err := src.Get([]byte(key))
		if err != nil {
			srcVal = nil
		}

		dstVal, err := dst.Get([]byte(key))
		if err != nil {
			dstVal = nil
		}

		if string(srcVal) != string(dstVal) {
			return fmt.Errorf("stable key %s mismatch: source=%q, destination=%q", key, srcVal, dstVal)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Fully stop the Raft node before migrating so term cannot change mid-copy.
2. Delete or empty the destination store so stale CurrentTerm/LastVoteTerm cannot survive.
3. Check that MigrateToWAL's copy loop persists both uint64 keys before verification.
4. Use the printed source/destination values to determine whether the destination is stale or the copy dropped the key.
5. Re-run the migration and confirm verifyMigration returns nil.

Example fix

// before
raftutil.MigrateToWAL(src, existingDstPath) // dst holds stale CurrentTerm from old node
// after
raft.Shutdown() // halt raft so term cannot change
os.Remove(existingDstPath)   // remove stale destination
raftutil.MigrateToWAL(src, existingDstPath)

// Before migrating, ensure the destination has no stale stable state:
for _, key := range []string{"CurrentTerm", "LastVoteTerm"} {
    if _, err := dst.GetUint64([]byte(key)); err == nil {
        return fmt.Errorf("destination already holds %s; use a fresh store", key)
    }
}

if err := raftutil.MigrateToWAL(src, dstPath); err != nil {
    if strings.Contains(err.Error(), "stable key") && strings.Contains(err.Error(), "mismatch") {
        // destination is stale or copy dropped a key; wipe and retry
        os.Remove(dstPath)
    }
    return err
}

When it happens

Trigger: After MigrateToWAL, dst.GetUint64(key) differs from src.GetUint64(key) for "CurrentTerm" or "LastVoteTerm" — the copy wrote a stale/zero value, the destination retained data from a previous use, or the node kept running and the term advanced between copy and verify.

Common situations: Migrating into a reused bolt DB holding state from an earlier node/cluster; running migration while the Raft node is live; a copy-loop bug skipping uint64 stable keys.

Related errors

• failed to copy stable store: %w
• data verification failed: %w
• failed to get destination uint64 key %s: %w
• stable key %s mismatch: source=%q, destination=%q
• failed to get destination log %d: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/5c5035259ada2889. Report an issue: GitHub.