hashicorp/nomad · error

failed to copy logs: %w

Error message

failed to copy logs: %w

What it means

migrate.CopyLogs copies all raft log entries from the BoltDB source into the new WAL store in ~64 MiB batches, respecting ctx cancellation. This error wraps any failure during that copy — a read failure from BoltDB, a write/flush failure into the WAL, or a context cancellation. On failure both stores are closed and the WAL directory is removed, leaving raft.db intact for retry.

Source

Thrown at helper/raftutil/migrate.go:109

		return fmt.Errorf("failed to create WAL directory: %w", err)
	}

	dst, err := raftwal.Open(walDir)
	if err != nil {
		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("failed to open WAL store: %w", err)
	}

	// Copy logs.
	logProgress := make(chan string, 64)
	wg.Add(1)
	go drainProgress(logProgress, progress, &wg)
	if err := migrate.CopyLogs(ctx, dst, src, migrateBatchBytes, logProgress); err != nil {
		dst.Close()
		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("failed to copy logs: %w", err)
	}

	// Copy stable store keys.
	stableProgress := make(chan string, 64)
	wg.Add(1)
	go drainProgress(stableProgress, progress, &wg)
	if err := migrate.CopyStable(ctx, dst, src, nil, nil, stableProgress); err != nil {
		dst.Close()
		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("failed to copy stable store: %w", err)
	}

	// Verify data integrity before finalizing.
	sendProgress(progress, "verifying migrated data")
	if err := verifyMigration(src, dst); err != nil {
		dst.Close()
		src.Close()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped error — for context cancellation, re-run with a longer-lived context and no artificial deadline.
2. Free disk space: the copy temporarily needs space for both formats (bolt size + 512 MiB buffer at minimum).
3. Verify BoltDB integrity (bolt check on raft.db) if the wrapped error indicates a read/corruption failure.
4. Re-run MigrateToWAL — it is safe to retry because the failed WAL directory is cleaned up automatically.

if boltFile, err := os.Open(filepath.Join(raftDir, "raft.db")); err == nil {
    boltFile.Close()
}
// also ensure free space for the WAL copy:
// free space >= raft.db size + 512 MiB

ctx, cancel := context.WithTimeout(context.Background(), maxMigrationDuration)
defer cancel()
err := raftutil.MigrateToWAL(ctx, raftDir, progress)
if err != nil {
    if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
        // retry with a longer window; source raft.db is untouched
    }
}

When it happens

Trigger: migrate.CopyLogs(ctx, dst, src, migrateBatchBytes, logProgress) returns an error: corrupted BoltDB log entries, disk full while writing WAL segments, I/O errors on either store, or ctx cancelled/expired mid-copy.

Common situations: Disk runs out during the copy of a large raft.db; operator cancels the context or the migration command times out on a very large log; BoltDB file corrupted by a previous crash; flaky network storage drops I/O mid-copy.

Related errors

• failed to copy stable store: %w
• failed to read snapshot: %w
• failed to open BoltDB store: %w
• data verification failed: %w
• migration succeeded but failed to rename %s to %s: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/47603729e9a6fdab. Report an issue: GitHub.