hashicorp/nomad · critical

data verification failed: %w

Error message

data verification failed: %w

What it means

After copying logs and stable data, verifyMigration compares first/last log indexes, key stable-store values, and spot-checks sample log entries between source and destination. This error wraps any verification mismatch or read failure, meaning the migrated WAL copy is not provably identical to the BoltDB source. The migration is aborted and the WAL directory removed, preserving raft.db.

Source

Thrown at helper/raftutil/migrate.go:129

	// Copy stable store keys.
	stableProgress := make(chan string, 64)
	wg.Add(1)
	go drainProgress(stableProgress, progress, &wg)
	if err := migrate.CopyStable(ctx, dst, src, nil, nil, stableProgress); err != nil {
		dst.Close()
		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("failed to copy stable store: %w", err)
	}

	// Verify data integrity before finalizing.
	sendProgress(progress, "verifying migrated data")
	if err := verifyMigration(src, dst); err != nil {
		dst.Close()
		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("data verification failed: %w", err)
	}

	// Close both stores before renaming files.
	if err := dst.Close(); err != nil {
		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("failed to close WAL store: %w", err)
	}

	if err := src.Close(); err != nil {
		cleanupWAL(walDir)
		return fmt.Errorf("failed to close BoltDB store: %w", err)
	}

	// Rename the old BoltDB file to preserve it as a backup with timestamp.
	timestamp := time.Now().Format("20060102-150405")
	backupPath := fmt.Sprintf("%s.migrated.%s", boltPath, timestamp)
	if err := os.Rename(boltPath, backupPath); err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Confirm the Nomad server is fully stopped before migrating — concurrent writes are the leading cause of verification mismatches.
2. Read the wrapped error to identify which check failed (index vs stable key vs log entry) and compare against raft.db.
3. Re-run MigrateToWAL from a clean state after stopping all writers.
4. If mismatches persist on a healthy quiesced cluster, check storage integrity and raft-wal/raft-boltdb versions.

Example fix

// before
systemctl start nomad   # server restarted too early
// after
systemctl stop nomad && ./nomad operator raft migrate-wal && systemctl start nomad

// before migrating, guarantee no writers:
//   systemctl stop nomad (all server nodes)
//   lsof <raftDir>/raft.db  -> expect no output
if out, err := exec.Command("lsof", filepath.Join(raftDir, "raft.db")).Output(); err == nil && len(out) > 0 {
    return fmt.Errorf("raft.db is still open by another process; stop the server first")
}

err := raftutil.MigrateToWAL(ctx, raftDir, progress)
if err != nil && strings.Contains(err.Error(), "data verification failed") {
    // do NOT start the server on the wal directory;
    // raft.db is intact — investigate and retry after quiescing writers
}

When it happens

Trigger: verifyMigration(src, dst) returns an error: first/last index mismatch between stores, CurrentTerm/LastVoteTerm/LastVoteCand mismatch, a sample log entry (first, middle, last) differing in index/term/type/data, or an error reading indexes/logs from either store.

Common situations: Concurrent writes to raft.db during migration (the Nomad server was still running); storage silently corrupting data; a bug or interruption that left the WAL copy incomplete despite CopyLogs/CopyStable returning nil.

Related errors

• failed to get destination uint64 key %s: %w
• stable key %s mismatch: source=%d, destination=%d
• failed to get destination log %d: %w
• log entry %d mismatch
• failed to open BoltDB store: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/3b8ef9fd94b87074. Report an issue: GitHub.