apache/hadoop · error · PathCommitException

Unknown committer: "%s"

Error message

Unknown committer: "%s"

What it means

PathCommitException from S3ACommitterFactory when the configured committer name does not match any of the four factories: directory, partitioned, magic, or staging. The factory switch maps each name to a specific committer factory class, and any other token falls through to the default branch. The name is matched case-sensitively after trimming, so spelling must be exact.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/S3ACommitterFactory.java:139

    switch (name) {
    case COMMITTER_NAME_FILE:
    case "":
      factory = null;
      break;
    case COMMITTER_NAME_DIRECTORY:
      factory = new DirectoryStagingCommitterFactory();
      break;
    case COMMITTER_NAME_PARTITIONED:
      factory = new PartitionedStagingCommitterFactory();
      break;
    case COMMITTER_NAME_MAGIC:
      factory = new MagicS3GuardCommitterFactory();
      break;
    case InternalCommitterConstants.COMMITTER_NAME_STAGING:
      factory = new StagingCommitterFactory();
      break;
    default:
      throw new PathCommitException(outputPath,
          "Unknown committer: \"" + name + "\"");
    }
    return factory;
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Set fs.s3a.committer.name to one of: directory, partitioned, magic, staging
2. Double-check spelling and case (lowercase, no quotes, no whitespace)
3. Verify the value on the actually-running job config (print conf.get("fs.s3a.committer.name") or check the job XML) rather than only the source file

Example fix

# before
spark.hadoop.fs.s3a.committer.name=Staging

# after
spark.hadoop.fs.s3a.committer.name=staging

Set<String> valid = new HashSet<>(Arrays.asList(
    "directory", "partitioned", "magic", "staging"));
String name = conf.getTrimmed("fs.s3a.committer.name");
if (!valid.contains(name)) {
  throw new IOException("Invalid fs.s3a.committer.name='" + name
      + "'; valid: " + valid);
}

try {
  OutputCommitter committer = S3ACommitterFactory.createCommitter(factoryConf, outputPath);
} catch (PathCommitException e) {
  // unknown committer name: check the job's effective config and resubmit
  LOG.error("fs.s3a.committer.name invalid: {}", e.getMessage(), e);
}

When it happens

Trigger: S3ACommitterFactory.createCommitter (invoked from the committer framework during job or task setup) with fs.s3a.committer.name set to an unknown value such as 'files', 'mutation', 'newStaging' or an empty string.

Common situations: Copy-pasted documentation examples with wrong names; setting the old experimental name instead of the released one; Spark jobs setting spark.hadoop.fs.s3a.committer.name to a value not supported by the hadoop-aws version in use; name left empty after a config migration.

Related errors

• Multipart IO request {sdkRequest} rejected {header}
• Multipart uploads are disabled for the FileSystem, the commi
• Task attempt {attemptID} has a self-generated job UUID
• Filesystem not supported by this committer
• Unsupported block buffer "{name}"

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c0ccd5e289d44781. Report an issue: GitHub.