{"record":{"id":"ec3cae4470108781","repo":"hashicorp/nomad","slug":"failed-to-clean-task-directories-after-failed-migr","errorCode":null,"errorMessage":"failed to clean task directories after failed migration: %v","messagePattern":"failed to clean task directories after failed migration: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/migrate_hook.go","lineNumber":65,"sourceCode":"\n\t// Wait for a previous alloc - if any - to terminate\n\tif err := h.allocWatcher.Wait(ctx); err != nil {\n\t\treturn err\n\t}\n\n\t// Wait for data to be migrated from a previous alloc if applicable\n\tif err := h.allocWatcher.Migrate(ctx, h.allocDir); err != nil {\n\t\tif err == context.Canceled {\n\t\t\treturn err\n\t\t}\n\n\t\t// Soft-fail on migration errors\n\t\th.logger.Warn(\"error migrating data from previous alloc\", \"error\", err)\n\n\t\t// Recreate alloc dir to ensure a clean slate\n\t\th.allocDir.Destroy()\n\t\tif err := h.allocDir.Build(); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to clean task directories after failed migration: %v\", err)\n\t\t}\n\t}\n\n\treturn nil\n}\n","sourceCodeStart":47,"sourceCodeEnd":71,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/migrate_hook.go#L47-L71","documentation":"In migrate_hook's Prerun, if migrating data from a previous allocation fails, the hook soft-fails the migration but tries to destroy and rebuild the allocation directory to guarantee a clean slate. If that rebuild (allocDir.Destroy + allocDir.Build) also fails, Prerun returns this error and the allocation cannot start.","triggerScenarios":"Migration from a previous alloc errors (logged as a warning), then either allocDir.Destroy() or allocDir.Build() returns an error when recreating the alloc directory tree.","commonSituations":"Disk full or permission problems on host_volume/data_dir paths; leftover directory locks or stale bind mounts from crashed tasks; filesystem errors on the Nomad client's data_dir; chroot/build template failures due to missing directories.","solutions":["Check the Nomad client log for the preceding 'error migrating data' warning and the underlying destroy/build failure.","Free disk space and fix permissions on the client's data_dir / alloc directory.","Stop stale tasks and manually remove the leftover alloc directory under data_dir/alloc, then let Nomad recreate it.","Restart the Nomad client to release stale mounts/locks, then allow the allocation to reschedule."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// pre-flight: ensure data_dir has space and writable alloc dirs\nif err := os.MkdirAll(allocDirPath, 0o755); err != nil {\n    return fmt.Errorf(\"data_dir not writable: %w\", err)\n}","typeGuard":"func canRebuildAllocDir(path string) bool {\n    st, err := os.Stat(path)\n    return err == nil && st.IsDir() && unix.Access(path, unix.W_OK) == nil\n}","tryCatchPattern":"if err := hook.Prerun(); err != nil && strings.Contains(err.Error(), \"failed to clean task directories\") {\n    log.Printf(\"alloc dir rebuild failed: %v — check disk space/permissions on data_dir\", err)\n}","preventionTips":["Monitor free disk space on Nomad clients (fingerprint reserves disk).","Run the nomad agent with a user that owns data_dir.","Clean leftover alloc dirs after crashed tasks.","Avoid stale bind mounts; restart the agent after hard host failures."],"tags":["nomad","allocdir","migration","filesystem"],"backgroundTag":"alloc-dir-build-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}