apache/hadoop · error · PathCommitException

%s: unknown conflict resolution mode: %s

Error message

%s: unknown conflict resolution mode: %s

What it means

PathCommitException from PartitionedStagingCommitter's pre-commit (cleanup) switch when getConflictResolutionMode returns a ConflictResolution constant the switch does not handle. As in DirectoryStagingCommitter, stock code parses the config via ConflictResolution.valueOf, so a bad fs.s3a.committer.staging.conflict-mode string throws IllegalArgumentException before this point; hitting the default branch means the runtime enum has a constant this switch was never taught (custom build or version skew). Unlike its directory sibling this is thrown as PathCommitException with an empty path argument.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/PartitionedStagingCommitter.java:154

    // enforce conflict resolution
    Configuration fsConf = fs.getConf();
    boolean shouldPrecheckPendingFiles = true;
    switch (getConflictResolutionMode(commitContext.getJobContext(), fsConf)) {
    case FAIL:
      // FAIL checking is done on the task side, so this does nothing
      break;
    case APPEND:
      // no check is needed because the output may exist for appending
      break;
    case REPLACE:
      // identify and replace the destination partitions
      replacePartitions(commitContext, pending);
      // and so there is no need to do another check.
      shouldPrecheckPendingFiles = false;
      break;
    default:
      throw new PathCommitException("",
          getRole() + ": unknown conflict resolution mode: "
          + getConflictResolutionMode(commitContext.getJobContext(), fsConf));
    }
    if (shouldPrecheckPendingFiles) {
      precommitCheckPendingFiles(commitContext, pending);
    }
  }

  /**
   * Identify all partitions which need to be replaced and then delete them.
   * The original implementation relied on all the pending commits to be
   * loaded so could simply enumerate them.
   * This iteration does not do that; it has to reload all the files
   * to build the set, after which it initiates the delete process.
   * This is done in parallel.
   * <pre>
   *   Set<Path> partitions = pending.stream()
   *     .map(Path::getParent)

View on GitHub (pinned to 2add963021)

Solutions

1. Use matching, released hadoop-aws/hadoop-common versions everywhere
2. In forks that add conflict modes, extend this switch (and the DirectoryStagingCommitter one) with the new case
3. Confirm fs.s3a.committer.staging.conflict-mode is one of fail, append, replace

String mode = fsConf.getTrimmed("fs.s3a.committer.staging.conflict-mode", "append");
if (!Arrays.asList("fail", "append", "replace").contains(mode)) {
  throw new IOException("Invalid conflict mode: " + mode);
}

try {
  ((PartitionedStagingCommitter) committer).cleanup(context, pending);
} catch (PathCommitException e) {
  if (String.valueOf(e.getMessage()).contains("unknown conflict resolution mode")) {
    // enum/switch skew between hadoop-aws jars -- align versions and rerun the job
    LOG.error("Version skew suspected in partitioned staging committer", e);
  }
}

When it happens

Trigger: Partitioned staging committer encounters an extended ConflictResolution constant from a patched or mismatched hadoop-aws jar during cleanup of replaced partitions.

Common situations: In-house hadoop-aws fork adds a conflict mode without updating PartitionedStagingCommitter; mixed Hadoop versions in a shared cluster classpath.

Related errors

• %s: unknown conflict resolution mode: %s
• E_DEST_EXISTS
• Multipart IO request {sdkRequest} rejected {header}
• Multipart uploads are disabled for the FileSystem, the commi
• Unable to recover task %s

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bacc4cfa7b2783d7. Report an issue: GitHub.