hashicorp/nomad · error

failed to get source log %d: %w

Error message

failed to get source log %d: %w

What it means

During the spot-check phase of verifyMigration (migrate.go:330), reading a log entry at a sampled index (first, middle, or last) from the SOURCE LogStore via GetLog failed. The source is authoritative, so this means the source itself is unreadable or corrupt and the migration cannot be verified; the index and wrapped cause are returned.

Source

Thrown at helper/raftutil/migrate.go:330

			return fmt.Errorf("stable key %s mismatch: source=%q, destination=%q", key, srcVal, dstVal)
		}
	}

	// If we have logs, spot-check a few entries for data integrity.
	if srcFirst > 0 && srcLast > 0 {
		indicesToCheck := []uint64{srcFirst}
		if srcLast > srcFirst {
			// Check middle entry.
			middle := srcFirst + (srcLast-srcFirst)/2
			indicesToCheck = append(indicesToCheck, middle)
			// Check last entry.
			indicesToCheck = append(indicesToCheck, srcLast)
		}

		for _, idx := range indicesToCheck {
			var srcLog, dstLog raft.Log
			if err := src.GetLog(idx, &srcLog); err != nil {
				return fmt.Errorf("failed to get source log %d: %w", idx, err)
			}
			if err := dst.GetLog(idx, &dstLog); err != nil {
				return fmt.Errorf("failed to get destination log %d: %w", idx, err)
			}

			if srcLog.Index != dstLog.Index || srcLog.Term != dstLog.Term ||
				srcLog.Type != dstLog.Type || string(srcLog.Data) != string(dstLog.Data) {
				return fmt.Errorf("log entry %d mismatch", idx)
			}
		}
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped error to identify the underlying read failure.
2. Check source store integrity (bolt integrity check, file permissions, disk health).
3. Reconcile srcFirst/srcLast with what GetLog can actually return; restore from backup if corrupt.
4. If compaction legitimately removed entries, fix the source's index reporting before migrating.
5. Retry the migration after repairing or replacing the source store.

Example fix

// before
raftutil.MigrateToWAL(corruptSrc, dstPath) // srcLast points at unreadable entry
// after
if err := boltCheck(srcPath); err != nil {
    src = restoreFromBackup(srcPath) // repair source first
}
raftutil.MigrateToWAL(src, dstPath)

// Sanity-check the source is fully readable before migrating:
first, _ := src.FirstIndex()
last, _ := src.LastIndex()
for _, idx := range []uint64{first, first + (last-first)/2, last} {
    var l raft.Log
    if err := src.GetLog(idx, &l); err != nil {
        return fmt.Errorf("source unreadable at %d; repair/restore before migrating: %w", idx, err)
    }
}

if err := raftutil.MigrateToWAL(src, dstPath); err != nil {
    if strings.Contains(err.Error(), "failed to get source log") {
        // source corrupt: restore from backup instead of retrying
        src = restoreFromBackup(srcPath)
    }
    return err
}

When it happens

Trigger: src.GetLog(idx, &srcLog) errors for idx in {srcFirst, middle, srcLast} after index-range checks passed — the source's index metadata says the entry exists but it is corrupt/truncated, the source file is damaged, or an IO error occurs during read.

Common situations: Corrupted raft.db or log file after a crash or disk failure; a source implementation whose LastIndex reports entries GetLog cannot return (stale index); log compaction removed entries the source still advertises.

Related errors

• failed to get destination log %d: %w
• failed to read log entry at index %d: %v
• failed to open BoltDB store: %w
• failed to copy logs: %w
• failed to copy stable store: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6380310f11fe1bc1. Report an issue: GitHub.