{"record":{"id":"d0f96660cf7a2528","repo":"apache/hadoop","slug":"invoking-cleanuppartialoutputfortask-from-non-p","errorCode":null,"errorMessage":"Invoking cleanUpPartialOutputForTask() from non @Preemptable class","messagePattern":"Invoking cleanUpPartialOutputForTask\\(\\) from non @Preemptable class","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialFileOutputCommitter.java","lineNumber":83,"sourceCode":"        String.valueOf(context.getTaskAttemptID()));\n  }\n\n  @VisibleForTesting\n  FileSystem fsFor(Path p, Configuration conf) throws IOException {\n    return p.getFileSystem(conf);\n  }\n\n  @Override\n  public void cleanUpPartialOutputForTask(TaskAttemptContext context)\n      throws IOException {\n\n    // we double check this is never invoked from a non-preemptable subclass.\n    // This should never happen, since the invoking codes is checking it too,\n    // but it is safer to double check. Errors handling this would produce\n    // inconsistent output.\n\n    if (!this.getClass().isAnnotationPresent(Checkpointable.class)) {\n      throw new IllegalStateException(\"Invoking cleanUpPartialOutputForTask() \" +\n          \"from non @Preemptable class\");\n    }\n    FileSystem fs =\n      fsFor(getTaskAttemptPath(context), context.getConfiguration());\n\n    LOG.info(\"cleanUpPartialOutputForTask: removing everything belonging to \" +\n        context.getTaskAttemptID().getTaskID() + \" in: \" +\n        getCommittedTaskPath(context).getParent());\n\n    final TaskAttemptID taid = context.getTaskAttemptID();\n    final TaskID tid = taid.getTaskID();\n    Path pCommit = getCommittedTaskPath(context).getParent();\n    // remove any committed output\n    for (int i = 0; i < taid.getId(); ++i) {\n      TaskAttemptID oldId = new TaskAttemptID(tid, i);\n      Path pTask = new Path(pCommit, oldId.toString());\n      if (!fs.delete(pTask, true) && fs.exists(pTask)) {\n        throw new IOException(\"Failed to delete \" + pTask);","sourceCodeStart":65,"sourceCodeEnd":101,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialFileOutputCommitter.java#L65-L101","documentation":"PartialFileOutputCommitter.cleanUpPartialOutputForTask() deletes partial output of preempted task attempts, and is only legal for committers marked as checkpoint/preemption-capable. As a belt-and-braces guard the method checks that the concrete class carries the @Checkpointable annotation and throws IllegalStateException otherwise (the message still says '@Preemptable', the annotation's old name). This is a programming-contract violation, not an environmental fault - invoking it on a non-checkpointable committer would leave inconsistent output.","triggerScenarios":"Calling cleanUpPartialOutputForTask(context) on a PartialFileOutputCommitter subclass whose class is not annotated @Checkpointable - typically a custom subclass, or direct invocation from application/test code instead of the preemption framework path.","commonSituations":"Extending PartialFileOutputCommitter for a custom layout and forgetting the annotation; refactoring that drops the annotation; test code calling the API directly on the base or a plain subclass.","solutions":["Annotate your committer class with @org.apache.hadoop.mapreduce.Checkpointable (the message text '@Preemptable' refers to this same annotation by its legacy name).","If your committer is not meant to support preemption, stop calling cleanUpPartialOutputForTask from your code path.","Rebuild and redeploy so the annotated class is what the task actually loads."],"exampleFix":"// before\nclass MyPartialCommitter extends PartialFileOutputCommitter {\n  public MyPartialCommitter(Path out, TaskAttemptContext ctx) throws IOException { super(out, ctx); }\n}\n\n// after\n@Checkpointable\nclass MyPartialCommitter extends PartialFileOutputCommitter {\n  public MyPartialCommitter(Path out, TaskAttemptContext ctx) throws IOException { super(out, ctx); }\n}","handlingStrategy":"type-guard","validationCode":null,"typeGuard":"public static boolean isCheckpointableCommitter(OutputCommitter c) {\n  return c.getClass().isAnnotationPresent(Checkpointable.class);\n}","tryCatchPattern":"if (!isCheckpointableCommitter(committer)) {\n  throw new IllegalStateException(\n      \"cleanUpPartialOutputForTask requires @Checkpointable on \" + committer.getClass().getName());\n}\ncommitter.cleanUpPartialOutputForTask(context);","preventionTips":["Annotate every custom PartialFileOutputCommitter subclass with @Checkpointable at class level.","Never call cleanUpPartialOutputForTask outside the preemption/checkpoint framework path.","Add a reflection-based test asserting the annotation survives refactors."],"tags":["hadoop","mapreduce","preemption","checkpointing","annotations","output-committer"],"backgroundTag":"missing-annotation-check","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}