{"record":{"id":"6608f322ff9ffee0","repo":"apache/hadoop","slug":"failed-to-delete-ptask","errorCode":null,"errorMessage":"Failed to delete {pTask}","messagePattern":"Failed to delete (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialFileOutputCommitter.java","lineNumber":101,"sourceCode":"      throw new IllegalStateException(\"Invoking cleanUpPartialOutputForTask() \" +\n          \"from non @Preemptable class\");\n    }\n    FileSystem fs =\n      fsFor(getTaskAttemptPath(context), context.getConfiguration());\n\n    LOG.info(\"cleanUpPartialOutputForTask: removing everything belonging to \" +\n        context.getTaskAttemptID().getTaskID() + \" in: \" +\n        getCommittedTaskPath(context).getParent());\n\n    final TaskAttemptID taid = context.getTaskAttemptID();\n    final TaskID tid = taid.getTaskID();\n    Path pCommit = getCommittedTaskPath(context).getParent();\n    // remove any committed output\n    for (int i = 0; i < taid.getId(); ++i) {\n      TaskAttemptID oldId = new TaskAttemptID(tid, i);\n      Path pTask = new Path(pCommit, oldId.toString());\n      if (!fs.delete(pTask, true) && fs.exists(pTask)) {\n        throw new IOException(\"Failed to delete \" + pTask);\n      }\n    }\n  }\n\n}\n","sourceCodeStart":83,"sourceCodeEnd":107,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialFileOutputCommitter.java#L83-L107","documentation":"During cleanUpPartialOutputForTask, the committer loops over earlier task-attempt directories (attempt 0..N-1 under the committed task path) and deletes each one. If FileSystem.delete(path, recursive=true) returns false AND the path still exists afterwards, it throws IOException('Failed to delete <path>') because a stale committed attempt could otherwise be double-committed later. This indicates the filesystem refused or partially performed the delete.","triggerScenarios":"fs.delete(pTask, true) returning false while fs.exists(pTask) is true, for a previously committed attempt directory: no write/parent permission on the output tree, HDFS in safemode, a concurrent cleaner racing the delete, or a store returning inconsistent results.","commonSituations":"Output directory owned by a different user than the one running cleanup (e.g. yarn vs. submitter after a manual mkdir); HDFS safemode or namemode failover mid-operation; external scripts pruning the same directories; NFS/mount flakiness on the staging area.","solutions":["Check permissions and ownership of the committed-task parent directory (the directory printed via getCommittedTaskPath(context).getParent()) and fix with chown/chmod.","Verify the filesystem is healthy (out of safemode, no failover in progress) and retry the preemption cleanup.","Look for a concurrent process deleting the same paths; if one exists, remove the race or accept that delete can return false because the path is already gone.","As a last resort, manually remove the stale attempt directory named in the message and rerun."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before cleanup: prove the output tree is deletable by this user\nPath parent = FileOutputCommitter.getCommittedTaskPath(context).getParent();\nFileSystem fs = parent.getFileSystem(conf);\nif (!fs.getFileStatus(parent).getPermission()\n        .getUserAction().implies(FsAction.WRITE)) {\n  throw new IOException(\"No write permission on \" + parent + \" - fix before cleanup\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  committer.cleanUpPartialOutputForTask(context);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Failed to delete\")) {\n    // transient FS state: verify health, fix permissions, then retry once\n  }\n}","preventionTips":["Create output directories as the user the job runs as, or pre-chown them to that user.","Do not run external cleanup scripts over committer-managed directories while tasks run.","Treat delete-false-and-exists as a hard integrity error for this committer - never ignore and continue."],"tags":["hadoop","mapreduce","filesystem","delete","permissions","cleanup","preemption"],"backgroundTag":"filesystem-delete-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}