What causes "failed to get destination log %d: %w" in hashicorp/nomad?

Check the wrapped error for the underlying destination failure (disk full, permissions, corruption).

failed to get destination log %d: %w

Error message

failed to get destination log %d: %w

What it means

During the spot-check phase of verifyMigration (migrate.go:333), reading a log entry at a sampled index (first, middle, or last) from the DESTINATION LogStore via GetLog failed. The migrated destination must return every entry the source can; a read failure means incomplete or unreadable data was written, so the migration aborts with the index and wrapped cause.

Source

Thrown at helper/raftutil/migrate.go:333

	// If we have logs, spot-check a few entries for data integrity.
	if srcFirst > 0 && srcLast > 0 {
		indicesToCheck := []uint64{srcFirst}
		if srcLast > srcFirst {
			// Check middle entry.
			middle := srcFirst + (srcLast-srcFirst)/2
			indicesToCheck = append(indicesToCheck, middle)
			// Check last entry.
			indicesToCheck = append(indicesToCheck, srcLast)
		}

		for _, idx := range indicesToCheck {
			var srcLog, dstLog raft.Log
			if err := src.GetLog(idx, &srcLog); err != nil {
				return fmt.Errorf("failed to get source log %d: %w", idx, err)
			}
			if err := dst.GetLog(idx, &dstLog); err != nil {
				return fmt.Errorf("failed to get destination log %d: %w", idx, err)
			}

			if srcLog.Index != dstLog.Index || srcLog.Term != dstLog.Term ||
				srcLog.Type != dstLog.Type || string(srcLog.Data) != string(dstLog.Data) {
				return fmt.Errorf("log entry %d mismatch", idx)
			}
		}
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Check the wrapped error for the underlying destination failure (disk full, permissions, corruption).
Verify MigrateToWAL copied every entry from srcFirst..srcLast and committed its transaction before verification.
Re-run the migration on a fresh destination path to rule out a partially written DB.
Close/flush the destination store cleanly after copy, then reopen it for verification.
Check destination disk space and file permissions before re-running.

Example fix

// before
raftutil.MigrateToWAL(src, dstPath)
verifyMigration(src, dst) // dst.GetLog fails: copy not flushed
// after
if err := raftutil.MigrateToWAL(src, dstPath); err != nil { return err }
dst.Close() // ensure bolt tx committed and flushed
dst, _ = raftboltdb.NewBoltStore(dstPath)
verifyMigration(src, dst)

Defensive patterns

Strategy: validation

Validate before calling

// Spot-check the destination is readable right after the copy, before full verification:
first, _ := dst.FirstIndex()
last, _ := dst.LastIndex()
var l raft.Log
if first > 0 {
    if err := dst.GetLog(last, &l); err != nil {
        return fmt.Errorf("destination unreadable at %d after copy; disk/commit issue: %w", last, err)
    }
}

Try / catch

if err := raftutil.MigrateToWAL(src, dstPath); err != nil {
    if strings.Contains(err.Error(), "failed to get destination log") {
        log.Printf("destination copy incomplete: %v (cause: %v)", err, errors.Unwrap(err))
        os.Remove(dstPath) // drop the bad copy and retry fresh
    }
    return err
}

Prevention

Ensure disk has enough free space for the full log before migrating.
Close/sync the destination bolt store cleanly after the copy, then reopen for verification.
Never kill the process mid-migration; run it under a supervisor with completion checks.
Verify the copy loop iterates srcFirst..srcLast inclusively with no skipped indices.
Check destination file permissions are writable by the migrating process.

When it happens

Trigger: dst.GetLog(idx, &dstLog) errors for a sampled idx after copy — the copy loop skipped or partially wrote entries, the destination bolt transaction was never committed/flushed, or the destination file is corrupt or on a full disk.

Common situations: Destination disk full or permission denied during migration; process killed mid-migration leaving an unsynced bolt DB; destination implementation returning errors for entries never inserted due to a copy-loop bug.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ad5b9ccf14f33264. Report an issue: GitHub.