apache/hadoop · error · IllegalStateException

Invoking cleanUpPartialOutputForTask() from non @Preemptable

Error message

Invoking cleanUpPartialOutputForTask() from non @Preemptable class

What it means

PartialFileOutputCommitter.cleanUpPartialOutputForTask() deletes partial output of preempted task attempts, and is only legal for committers marked as checkpoint/preemption-capable. As a belt-and-braces guard the method checks that the concrete class carries the @Checkpointable annotation and throws IllegalStateException otherwise (the message still says '@Preemptable', the annotation's old name). This is a programming-contract violation, not an environmental fault - invoking it on a non-checkpointable committer would leave inconsistent output.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialFileOutputCommitter.java:83

        String.valueOf(context.getTaskAttemptID()));
  }

  @VisibleForTesting
  FileSystem fsFor(Path p, Configuration conf) throws IOException {
    return p.getFileSystem(conf);
  }

  @Override
  public void cleanUpPartialOutputForTask(TaskAttemptContext context)
      throws IOException {

    // we double check this is never invoked from a non-preemptable subclass.
    // This should never happen, since the invoking codes is checking it too,
    // but it is safer to double check. Errors handling this would produce
    // inconsistent output.

    if (!this.getClass().isAnnotationPresent(Checkpointable.class)) {
      throw new IllegalStateException("Invoking cleanUpPartialOutputForTask() " +
          "from non @Preemptable class");
    }
    FileSystem fs =
      fsFor(getTaskAttemptPath(context), context.getConfiguration());

    LOG.info("cleanUpPartialOutputForTask: removing everything belonging to " +
        context.getTaskAttemptID().getTaskID() + " in: " +
        getCommittedTaskPath(context).getParent());

    final TaskAttemptID taid = context.getTaskAttemptID();
    final TaskID tid = taid.getTaskID();
    Path pCommit = getCommittedTaskPath(context).getParent();
    // remove any committed output
    for (int i = 0; i < taid.getId(); ++i) {
      TaskAttemptID oldId = new TaskAttemptID(tid, i);
      Path pTask = new Path(pCommit, oldId.toString());
      if (!fs.delete(pTask, true) && fs.exists(pTask)) {
        throw new IOException("Failed to delete " + pTask);

View on GitHub (pinned to 2add963021)

Solutions

Annotate your committer class with @org.apache.hadoop.mapreduce.Checkpointable (the message text '@Preemptable' refers to this same annotation by its legacy name).
If your committer is not meant to support preemption, stop calling cleanUpPartialOutputForTask from your code path.
Rebuild and redeploy so the annotated class is what the task actually loads.

Example fix

// before
class MyPartialCommitter extends PartialFileOutputCommitter {
  public MyPartialCommitter(Path out, TaskAttemptContext ctx) throws IOException { super(out, ctx); }
}

// after
@Checkpointable
class MyPartialCommitter extends PartialFileOutputCommitter {
  public MyPartialCommitter(Path out, TaskAttemptContext ctx) throws IOException { super(out, ctx); }
}

Defensive patterns

Strategy: type-guard

Type guard

public static boolean isCheckpointableCommitter(OutputCommitter c) {
  return c.getClass().isAnnotationPresent(Checkpointable.class);
}

Try / catch

if (!isCheckpointableCommitter(committer)) {
  throw new IllegalStateException(
      "cleanUpPartialOutputForTask requires @Checkpointable on " + committer.getClass().getName());
}
committer.cleanUpPartialOutputForTask(context);

Prevention

Annotate every custom PartialFileOutputCommitter subclass with @Checkpointable at class level.
Never call cleanUpPartialOutputForTask outside the preemption/checkpoint framework path.
Add a reflection-based test asserting the annotation survives refactors.

When it happens

Trigger: Calling cleanUpPartialOutputForTask(context) on a PartialFileOutputCommitter subclass whose class is not annotated @Checkpointable - typically a custom subclass, or direct invocation from application/test code instead of the preemption framework path.

Common situations: Extending PartialFileOutputCommitter for a custom layout and forgetting the annotation; refactoring that drops the annotation; test code calling the API directly on the base or a plain subclass.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d0f96660cf7a2528. Report an issue: GitHub.