apache/hadoop · error · PathIOException

Failed to create Store Operations from configuration option

Error message

Failed to create Store Operations from configuration option mapreduce.manifest.committer.store.operations.classname:{e}

What it means

The manifest committer creates its store-operations layer from mapreduce.manifest.committer.store.operations.classname (default ManifestStoreOperationsThroughFileSystem): it loads the class, calls its no-arg constructor, then bindToFileSystem(fs, path). Any failure in that chain is wrapped in PathIOException('Failed to create Store Operations from configuration option ...:<cause>'). The nested exception is the actual story - wrong type, missing constructor, or bind-time validation.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/impl/ManifestCommitterSupport.java:290

   * @param path path under FS.
   * @return a bonded store operations.
   * @throws IOException on binding/init problems.
   */
  public static ManifestStoreOperations createManifestStoreOperations(
      final Configuration conf,
      final FileSystem filesystem,
      final Path path) throws IOException {
    try {
      final Class<? extends ManifestStoreOperations> storeClass = conf.getClass(
          OPT_STORE_OPERATIONS_CLASS,
          ManifestStoreOperationsThroughFileSystem.class,
          ManifestStoreOperations.class);
      final ManifestStoreOperations operations = storeClass.
          getDeclaredConstructor().newInstance();
      operations.bindToFileSystem(filesystem, path);
      return operations;
    } catch (Exception e) {
      throw new PathIOException(path.toString(),
          "Failed to create Store Operations from configuration option "
              + OPT_STORE_OPERATIONS_CLASS
              + ":" + e, e);
    }
  }

  /**
   * Logic to create directory names from job and attempt.
   * This is self-contained it so it can be used in tests
   * as well as in the committer.
   */
  public static class AttemptDirectories {

    /**
     * Job output path.
     */
    private final Path outputPath;

View on GitHub (pinned to 2add963021)

Solutions

1. Unwrap the cause (PathIOException.getCause()): ClassNotFoundException/NoSuchMethodException = packaging/constructor problem; other exceptions = bindToFileSystem rejected the filesystem/path.
2. Ensure the class is public, implements ManifestStoreOperations, and declares a public no-arg constructor; move any setup into bindToFileSystem().
3. Ship the class in the job jar and confirm it loads on both client and task nodes (identical version).
4. If the bind failed, verify the filesystem/path combination is what your implementation supports (scheme, authority, permissions).

Example fix

// before: only a constructor with arguments
public class MyStoreOps implements ManifestStoreOperations {
  public MyStoreOps(FileSystem fs) { }
}

// after: no-arg ctor + bind hook
public class MyStoreOps implements ManifestStoreOperations {
  public MyStoreOps() { }
  @Override public void bindToFileSystem(FileSystem fs, Path path) {
    // store fs/path here
  }
}

// verify the configured store-operations class is loadable and constructible
String cn = conf.get("mapreduce.manifest.committer.store.operations.classname", "");
if (!cn.isEmpty()) {
  Class<?> c = Class.forName(cn);
  Preconditions.checkState(ManifestStoreOperations.class.isAssignableFrom(c), "wrong type: " + cn);
  c.getDeclaredConstructor().newInstance(); // fails fast if no public no-arg ctor
}

try {
  ManifestStoreOperations ops = ManifestCommitterSupport.createStoreOperations(conf, fs, path);
} catch (PathIOException e) {
  Throwable cause = e.getCause() != null ? e.getCause() : e;
  // distinguish packaging/ctor problems (fix the class) from bind failures (fix the fs/path)
}

When it happens

Trigger: Configuring a custom ManifestStoreOperations class that (1) has no public no-arg constructor, (2) does not implement/extend ManifestStoreOperations (conf.getClass with that target type fails), or (3) whose bindToFileSystem() throws (unsupported filesystem, null filesystem, path validation).

Common situations: Plugging in a custom store-operations class for tuning or testing; class present at compile time but missing from the task-side jar; refactoring that adds constructor parameters; binding a store-operations implementation tied to a specific filesystem (e.g. ABFS) and then running against another FS.

Related errors

• Error creating plugin: {}
• No KeyProviderFactory for ${uri} in ${KEY_PROVIDER_PATH}
• Could not find configured fencing method {}
• Class {} does not implement FenceMethod
• Cannot initialize the class: {clazz}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/90e6d57a2760e3c8. Report an issue: GitHub.