apache/hadoop · error · UnsupportedOperationException

Resilient commit not supported

Error message

Resilient commit not supported

What it means

ManifestStoreOperations.commitFile() is the optional 'resilient commit' hook (commit a file through a store-native API instead of rename). The base implementation always throws UnsupportedOperationException('Resilient commit not supported'). The hook is only invoked when storeSupportsResilientCommit() returns true - the base returns false, and the ABFS implementation overrides both - so this exception means a subclass advertised resilient support but did not actually implement commitFile().

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/impl/ManifestStoreOperations.java:266

   * Commit one file through any resilient API.
   * This operation MUST rename source to destination,
   * else raise an exception.
   * The result indicates whether or not some
   * form of recovery took place.
   *
   * If etags were collected during task commit, these will be
   * in the entries passed in here.
   *
   * The base implementation always raises
   * {@code UnsupportedOperationException}
   * @param entry entry to commit
   * @return the result of the commit
   * @throws IOException failure.
   * @throws UnsupportedOperationException if not available.
   *
   */
  public CommitFileResult commitFile(FileEntry entry) throws IOException {
    throw new UnsupportedOperationException("Resilient commit not supported");
  }

  /**
   * Outcome from the operation {@link #commitFile(FileEntry)}.
   * As a rename failure MUST raise an exception, this result
   * only declares whether or not some form of recovery took place.
   */
  public static final class CommitFileResult {

    /** Did recovery take place? */
    private final boolean recovered;

    /** Time waiting for IO capacity, may be null. */
    @Nullable
    private final Duration waitTime;

    /**
     * Full commit result.

View on GitHub (pinned to 2add963021)

Solutions

1. If you do not implement a native commit API, make storeSupportsResilientCommit() return false so the committer uses the rename path.
2. If you need resilient commit, implement commitFile(FileEntry) to move source to dest or raise IOException - copy the pattern from the ABFS subclass of ManifestStoreOperationsThroughFileSystem.
3. Add a unit test asserting: storeSupportsResilientCommit()==true implies commitFile() completes on your store.

Example fix

// before: capability declared, hook missing
public class MyStoreOps extends ManifestStoreOperations {
  @Override public boolean storeSupportsResilientCommit() { return true; }
}

// after: keep the flag honest
public class MyStoreOps extends ManifestStoreOperations {
  @Override public boolean storeSupportsResilientCommit() { return false; }
  // OR implement: @Override public CommitFileResult commitFile(FileEntry e) { ... }
}

public static boolean resilientCommitIsImplemented(ManifestStoreOperations ops) {
  if (!ops.storeSupportsResilientCommit()) {
    return false; // base rename path will be used, safe
  }
  try {
    ops.getClass().getDeclaredMethod("commitFile", FileEntry.class);
    return true; // subclass actually overrides the hook
  } catch (NoSuchMethodException notOverridden) {
    return false; // advertising support without commitFile() would throw UOE
  }
}

try {
  CommitFileResult r = operations.commitFile(entry);
} catch (UnsupportedOperationException e) {
  // subclass contract violation: stop advertising resilient support or implement commitFile
}

When it happens

Trigger: A custom ManifestStoreOperations (set via mapreduce.manifest.committer.store.operations.classname) whose storeSupportsResilientCommit() returns true without overriding commitFile(FileEntry); during file commit, AbstractJobOrTaskStage.commitFile() then routes through operations.commitFile(entry) and hits the base throw.

Common situations: Copy-pasting the ABFS store-operations flag into a custom class without porting its commitFile(); partial refactors where the capability flag survives but the implementation is dropped.

Related errors

• {getClass().getSimpleName()} doesn't support truncate
• Append is not supported by ChecksumFileSystem
• Truncate is not supported by ChecksumFileSystem
• Concat is not supported by ChecksumFileSystem
• Truncate is not supported by ChecksumFs

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/edeea402a06934d6. Report an issue: GitHub.