apache/hadoop · error · PathCommitException

: unknown conflict resolution mode

Error message

%s: unknown conflict resolution mode: %s

What it means

PathCommitException from PartitionedStagingCommitter's pre-commit (cleanup) switch when getConflictResolutionMode returns a ConflictResolution constant the switch does not handle. As in DirectoryStagingCommitter, stock code parses the config via ConflictResolution.valueOf, so a bad fs.s3a.committer.staging.conflict-mode string throws IllegalArgumentException before this point; hitting the default branch means the runtime enum has a constant this switch was never taught (custom build or version skew). Unlike its directory sibling this is thrown as PathCommitException with an empty path argument.

Solutions

1. Use matching, released hadoop-aws/hadoop-common versions everywhere
2. In forks that add conflict modes, extend this switch (and the DirectoryStagingCommitter one) with the new case
3. Confirm fs.s3a.committer.staging.conflict-mode is one of fail, append, replace

String mode = fsConf.getTrimmed("fs.s3a.committer.staging.conflict-mode", "append");
if (!Arrays.asList("fail", "append", "replace").contains(mode)) {
  throw new IOException("Invalid conflict mode: " + mode);
}

try {
  ((PartitionedStagingCommitter) committer).cleanup(context, pending);
} catch (PathCommitException e) {
  if (String.valueOf(e.getMessage()).contains("unknown conflict resolution mode")) {
    // enum/switch skew between hadoop-aws jars -- align versions and rerun the job
    LOG.error("Version skew suspected in partitioned staging committer", e);
  }
}

When it happens

Trigger: Partitioned staging committer encounters an extended ConflictResolution constant from a patched or mismatched hadoop-aws jar during cleanup of replaced partitions.

Common situations: In-house hadoop-aws fork adds a conflict mode without updating PartitionedStagingCommitter; mixed Hadoop versions in a shared cluster classpath.

Related errors

• : unknown conflict resolution mode
• E_DEST_EXISTS
• Cannot parse URI
• E_NO_SPARK_UUID
• E_WRONG_FS

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bacc4cfa7b2783d7. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/PartitionedStagingCommitter.java:154

    // enforce conflict resolution
    Configuration fsConf = fs.getConf();
    boolean shouldPrecheckPendingFiles = true;
    switch (getConflictResolutionMode(commitContext.getJobContext(), fsConf)) {
    case FAIL:
      // FAIL checking is done on the task side, so this does nothing
      break;
    case APPEND:
      // no check is needed because the output may exist for appending
      break;
    case REPLACE:
      // identify and replace the destination partitions
      replacePartitions(commitContext, pending);
      // and so there is no need to do another check.
      shouldPrecheckPendingFiles = false;
      break;
    default:
      throw new PathCommitException("",
          getRole() + ": unknown conflict resolution mode: "
          + getConflictResolutionMode(commitContext.getJobContext(), fsConf));
    }
    if (shouldPrecheckPendingFiles) {
      precommitCheckPendingFiles(commitContext, pending);
    }
  }

  /**
   * Identify all partitions which need to be replaced and then delete them.
   * The original implementation relied on all the pending commits to be
   * loaded so could simply enumerate them.
   * This iteration does not do that; it has to reload all the files
   * to build the set, after which it initiates the delete process.
   * This is done in parallel.
   * <pre>
   *   Set<Path> partitions = pending.stream()
   *     .map(Path::getParent)

View on GitHub (pinned to 2add963021)