apache/hadoop · error · IOException

Failed to create {clazz}:{e}

Error message

Failed to create {clazz}:{e}

What it means

NamedCommitterFactory instantiates the class named by the config key mapreduce.outputcommitter.named.classname through its constructor taking (Path, TaskAttemptContext). If reflection cannot create the instance - missing constructor, abstract/interface class, non-public class or constructor, or the constructor itself throwing - the factory wraps the failure in IOException('Failed to create <clazz>:<cause>'). The real reason is always in the nested exception and in the ':'-separated tail of the message.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/NamedCommitterFactory.java:57

    PathOutputCommitterFactory {
  private static final Logger LOG =
      LoggerFactory.getLogger(NamedCommitterFactory.class);

  @SuppressWarnings("JavaReflectionMemberAccess")
  @Override
  public PathOutputCommitter createOutputCommitter(Path outputPath,
      TaskAttemptContext context) throws IOException {
    Class<? extends PathOutputCommitter> clazz = loadCommitterClass(context);
    LOG.debug("Using PathOutputCommitter implementation {}", clazz);
    try {
      Constructor<? extends PathOutputCommitter> ctor
          = clazz.getConstructor(Path.class, TaskAttemptContext.class);
      return ctor.newInstance(outputPath, context);
    } catch (NoSuchMethodException
        | InstantiationException
        | IllegalAccessException
        | InvocationTargetException e) {
      throw new IOException("Failed to create " + clazz
          + ":" + e, e);
    }
  }

  /**
   * Load the class named in {@link #NAMED_COMMITTER_CLASS}.
   * @param context job or task context
   * @return the committer class
   * @throws IOException if no committer was defined.
   */
  private Class<? extends PathOutputCommitter> loadCommitterClass(
      JobContext context) throws IOException {
    Preconditions.checkNotNull(context, "null context");
    Configuration conf = context.getConfiguration();
    String value = conf.get(NAMED_COMMITTER_CLASS, "");
    if (value.isEmpty()) {
      throw new IOException("No committer defined in " + NAMED_COMMITTER_CLASS);
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Read the nested cause: IOException.getCause() tells you whether it was NoSuchMethodException (wrong signature), InstantiationException (abstract), IllegalAccessException (not public/access) or InvocationTargetException (constructor threw - look one level deeper).
2. Give the committer class a public constructor 'public MyCommitter(Path outputPath, TaskAttemptContext context) throws IOException' that calls super(outputPath, context).
3. Make the class concrete and public, and ship it in the job jar so both client and tasks load the same version.
4. Unit-test instantiation directly: new MyCommitter(new Path("out"), new TaskAttemptContextImpl(conf, new TaskAttemptID())) before submitting.
5. If the constructor threw, fix the underlying error it reports (usually config or filesystem access), not the reflection path.

Example fix

// before: constructor has the wrong second parameter
class MyCommitter extends PathOutputCommitter {
  MyCommitter(Path out, JobContext ctx) throws IOException { super(out, ctx); }
}

// after: matches the signature NamedCommitterFactory looks up
public class MyCommitter extends PathOutputCommitter {
  public MyCommitter(Path outputPath, TaskAttemptContext context) throws IOException {
    super(outputPath, context);
  }
}

// before job submission: verify the named committer is constructible
String cn = conf.get("mapreduce.outputcommitter.named.classname", "");
Class<?> c = Class.forName(cn);
Preconditions.checkState(PathOutputCommitter.class.isAssignableFrom(c),
    "not a PathOutputCommitter: %s", cn);
c.getConstructor(Path.class, TaskAttemptContext.class); // throws NSME early, on the client

try {
  committerFactory.createOutputCommitter(outPath, context);
} catch (IOException e) {
  Throwable root = e.getCause() != null ? e.getCause() : e;
  if (root instanceof InvocationTargetException && root.getCause() != null) {
    root = root.getCause(); // the constructor's own failure
  }
  log.error("committer instantiation failed at root: {}", root, root);
  // fail the job: configuration error, retrying unchanged cannot help
}

When it happens

Trigger: Selecting the named factory (e.g. mapreduce.outputcommitter.factory.scheme.<scheme>=org.apache.hadoop.mapreduce.lib.output.NamedCommitterFactory) while the named class: (1) lacks a public constructor with signature (Path, TaskAttemptContext), (2) is abstract or an interface, (3) is not public or its constructor is not accessible, or (4) its constructor throws (e.g. NPE from reading config, FS errors).

Common situations: Writing a custom PathOutputCommitter with a (Path, JobContext) constructor instead of (Path, TaskAttemptContext); forgetting the public modifier; the committer class not being on the task classpath so a different/incompatible version is loaded; constructor doing filesystem work that fails at task startup.

Related errors

• Only 1 or 2 algorithm version is supported
• Unable to recover task %s, output: %s
• Error creating plugin: {}
• Failed to get method
• Invalid specification for distributed-cache artifacts of typ

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7ef605dcb6aaaa1b. Report an issue: GitHub.