What causes "failed to get destination uint64 key %s: %w" in hashicorp/nomad?

Inspect the wrapped error (errors.Unwrap) to find the real store failure (permissions, disk full, corruption).

failed to get destination uint64 key %s: %w

Error message

failed to get destination uint64 key %s: %w

What it means

This error is returned by verifyMigration in helper/raftutil/migrate.go:290 when reading a uint64 stable-store key (CurrentTerm or LastVoteTerm) from the DESTINATION store during a Raft store migration fails. The migration is aborted because the migrated store cannot read keys the source could read, so data integrity cannot be verified. The underlying store error is preserved via %w for diagnosis.

Source

Thrown at helper/raftutil/migrate.go:290

	if srcFirst != dstFirst {
		return fmt.Errorf("first index mismatch: source=%d, destination=%d", srcFirst, dstFirst)
	}

	if srcLast != dstLast {
		return fmt.Errorf("last index mismatch: source=%d, destination=%d", srcLast, dstLast)
	}

	// Verify stable store uint64 keys.
	uint64Keys := []string{"CurrentTerm", "LastVoteTerm"}
	for _, key := range uint64Keys {
		srcVal, err := src.GetUint64([]byte(key))
		if err != nil {
			return fmt.Errorf("failed to get source uint64 key %s: %w", key, err)
		}

		dstVal, err := dst.GetUint64([]byte(key))
		if err != nil {
			return fmt.Errorf("failed to get destination uint64 key %s: %w", key, err)
		}

		if srcVal != dstVal {
			return fmt.Errorf("stable key %s mismatch: source=%d, destination=%d", key, srcVal, dstVal)
		}
	}

	// Verify stable store byte keys.
	byteKeys := []string{"LastVoteCand"}
	for _, key := range byteKeys {
		srcVal, err := src.Get([]byte(key))
		if err != nil {
			srcVal = nil
		}

		dstVal, err := dst.Get([]byte(key))
		if err != nil {
			dstVal = nil

View on GitHub (pinned to 482b49bf1a)

Solutions

Inspect the wrapped error (errors.Unwrap) to find the real store failure (permissions, disk full, corruption).
Re-run the migration against a fresh, empty destination path so all stable keys get written before verification.
Verify MigrateToWAL's copy loop actually persists CurrentTerm and LastVoteTerm before calling verifyMigration.
Confirm the destination store implementation supports GetUint64 and is opened read-write.
Fix filesystem issues: permissions on the destination DB file and free disk space.

Example fix

// before
raftutil.MigrateToWAL(src, dstPath) // destination partially written, verify fails
// after
if err := os.RemoveAll(dstPath); err != nil { return err } // start clean
if err := raftutil.MigrateToWAL(src, dstPath); err != nil { return err }

Defensive patterns

Strategy: validation

Validate before calling

for _, key := range []string{"CurrentTerm", "LastVoteTerm"} {
    if _, err := dst.GetUint64([]byte(key)); err != nil {
        return fmt.Errorf("destination unreadable for %s before migration: %w", key, err)
    }
}

Try / catch

if err := raftutil.MigrateToWAL(src, dstPath); err != nil {
    log.Printf("migration failed: %v (cause: %v)", err, errors.Unwrap(err))
    return err
}

Prevention

Always migrate to a fresh, empty destination path.
Check destination file permissions and free disk space before migrating.
Stop the Raft node before migration so state is quiescent.
Inspect the wrapped %w cause immediately; it names the real store failure.
Write both uint64 stable keys to the destination and read them back as a smoke test before verification.

When it happens

Trigger: dst.GetUint64([]byte(key)) returns non-nil for "CurrentTerm" or "LastVoteTerm" while verifyMigration runs after MigrateToWAL — e.g. the copy loop never wrote the key, the destination bolt DB is corrupt, read-only, or not properly committed.

Common situations: Migrating into a destination file with wrong permissions or on a full disk; destination store from an older version lacking those keys; partially written destination from a previously failed migration; unit tests (TestVerifyMigration_IndexMismatch) with partially populated destinations.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/90ae859fdbe3b5c1. Report an issue: GitHub.