hashicorp/nomad · critical

log entry %d mismatch

Error message

log entry %d mismatch

What it means

verifyMigration spot-checks log entries (first, middle, last) after a Raft log migration to confirm the destination store holds identical entries to the source. It compares Index, Term, Type, and raw Data bytes; any difference returns "log entry %d mismatch". This is a data-integrity guard: the migrated log at that index is corrupt, truncated, or was written by an incompatible path.

Source

Thrown at helper/raftutil/migrate.go:338

			// Check middle entry.
			middle := srcFirst + (srcLast-srcFirst)/2
			indicesToCheck = append(indicesToCheck, middle)
			// Check last entry.
			indicesToCheck = append(indicesToCheck, srcLast)
		}

		for _, idx := range indicesToCheck {
			var srcLog, dstLog raft.Log
			if err := src.GetLog(idx, &srcLog); err != nil {
				return fmt.Errorf("failed to get source log %d: %w", idx, err)
			}
			if err := dst.GetLog(idx, &dstLog); err != nil {
				return fmt.Errorf("failed to get destination log %d: %w", idx, err)
			}

			if srcLog.Index != dstLog.Index || srcLog.Term != dstLog.Term ||
				srcLog.Type != dstLog.Type || string(srcLog.Data) != string(dstLog.Data) {
				return fmt.Errorf("log entry %d mismatch", idx)
			}
		}
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Delete the partially-migrated destination log store and re-run the migration from a clean, empty destination.
2. Dump both logs at the failing index (e.g. with raft-tools or GetLog) and compare Term/Type/Data to find whether entries shifted or were skipped during the copy.
3. Ensure the source store is not receiving new Raft appends during migration (stop the server or take a consistent snapshot first).
4. If the destination had pre-existing entries, wipe it before migrating so index 1..N map 1:1 with the source.

Example fix

// before: migrating into a destination that already had logs, causing index shift
err := raftutil.MigrateToWAL(boltStore, walStore) // "log entry 5 mismatch"
// after: start from an empty destination
os.RemoveAll(dstWALDir)
wal, _ := raftwal.NewWAL(...)
err := raftutil.MigrateToWAL(boltStore, wal) // clean 1:1 copy, verification passes

// Before migrating, confirm the destination is empty so indices map 1:1
first, last, err := dstStore.FirstIndex()/LastIndex()
if first != 0 || last != 0 {
    return fmt.Errorf("destination not empty (first=%d last=%d); wipe it before migration", first, last)
}

if err := raftutil.MigrateToWAL(src, dst); err != nil {
    var mismatchErr interface{ Error() string }
    if strings.Contains(err.Error(), "log entry ") && strings.Contains(err.Error(), "mismatch") {
        // integrity failure: do NOT trust the destination; wipe and re-migrate
        wipeDestination(dst)
        return retryMigration(src, dst)
    }
    return err
}

When it happens

Trigger: MigrateToWAL (or the test TestVerifyMigration_IndexMismatch) calls verifyMigration after copying a Raft log, and the destination log at the sampled index has a different Term, Type, Index, or Data payload than the source log at the same index.

Common situations: A partially-completed copy (destination truncated mid-migration); logs appended to the destination after the copy so indices shifted; an off-by-one or skipped-entry bug in the migration writer; restoring onto a store that already contained entries at those indices.

Related errors

• data verification failed: %w
• failed to get destination uint64 key %s: %w
• stable key %s mismatch: source=%d, destination=%d
• failed to get destination log %d: %w
• failed to open BoltDB store: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/4fcde260209dcb71. Report an issue: GitHub.