failed to clean task directories after failed migration: %v

Error message

failed to clean task directories after failed migration: %v

What it means

In migrate_hook's Prerun, if migrating data from a previous allocation fails, the hook soft-fails the migration but tries to destroy and rebuild the allocation directory to guarantee a clean slate. If that rebuild (allocDir.Destroy + allocDir.Build) also fails, Prerun returns this error and the allocation cannot start.

Source

Thrown at client/allocrunner/migrate_hook.go:65

	// Wait for a previous alloc - if any - to terminate
	if err := h.allocWatcher.Wait(ctx); err != nil {
		return err
	}

	// Wait for data to be migrated from a previous alloc if applicable
	if err := h.allocWatcher.Migrate(ctx, h.allocDir); err != nil {
		if err == context.Canceled {
			return err
		}

		// Soft-fail on migration errors
		h.logger.Warn("error migrating data from previous alloc", "error", err)

		// Recreate alloc dir to ensure a clean slate
		h.allocDir.Destroy()
		if err := h.allocDir.Build(); err != nil {
			return fmt.Errorf("failed to clean task directories after failed migration: %v", err)
		}
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Check the Nomad client log for the preceding 'error migrating data' warning and the underlying destroy/build failure.
Free disk space and fix permissions on the client's data_dir / alloc directory.
Stop stale tasks and manually remove the leftover alloc directory under data_dir/alloc, then let Nomad recreate it.
Restart the Nomad client to release stale mounts/locks, then allow the allocation to reschedule.

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-flight: ensure data_dir has space and writable alloc dirs
if err := os.MkdirAll(allocDirPath, 0o755); err != nil {
    return fmt.Errorf("data_dir not writable: %w", err)
}

Type guard

func canRebuildAllocDir(path string) bool {
    st, err := os.Stat(path)
    return err == nil && st.IsDir() && unix.Access(path, unix.W_OK) == nil
}

Try / catch

if err := hook.Prerun(); err != nil && strings.Contains(err.Error(), "failed to clean task directories") {
    log.Printf("alloc dir rebuild failed: %v — check disk space/permissions on data_dir", err)
}

Prevention

Monitor free disk space on Nomad clients (fingerprint reserves disk).
Run the nomad agent with a user that owns data_dir.
Clean leftover alloc dirs after crashed tasks.
Avoid stale bind mounts; restart the agent after hard host failures.

When it happens

Trigger: Migration from a previous alloc errors (logged as a warning), then either allocDir.Destroy() or allocDir.Build() returns an error when recreating the alloc directory tree.

Common situations: Disk full or permission problems on host_volume/data_dir paths; leftover directory locks or stale bind mounts from crashed tasks; filesystem errors on the Nomad client's data_dir; chroot/build template failures due to missing directories.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ec3cae4470108781. Report an issue: GitHub.