apache/hadoop · error · PathIOException

Conditional Writes Unavailable

Error message

Conditional Writes Unavailable

What it means

innerCreateFile() fails fast when the caller requested conditional creation (options isConditionalOverwrite()/isConditionalOverwriteEtag(), driven by the fs.option.create.conditional.overwrite / .overwrite.etag create options) but the filesystem was initialized with fs.s3a.create.conditional.enabled=false - the conditionalCreateEnabled field read once in initialize(). PathIOException('Conditional Writes Unavailable') is raised before any probe or PUT, because proceeding would silently drop the no-clobber guarantee the caller asked for.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2125

     This seems complicated, but comes down to
     "if explicitly requested and the FS enables it, use".
     */
    // create file attributes
    boolean cCreate = options.isConditionalOverwrite();
    boolean cEtag = options.isConditionalOverwriteEtag();
    boolean createPerf = options.isPerformance();
    boolean overwrite = flags.contains(CreateFlag.OVERWRITE);

    // path attributes
    boolean magic = isUnderMagicCommitPath(path);

    // store options
    // is CC available.
    boolean ccAvailable = conditionalCreateEnabled;

    if (!ccAvailable && (cCreate || cEtag)) {
      // fail fast if conditional creation is requested on an FS without it.
      throw new PathIOException(path.toString(), "Conditional Writes Unavailable");
    }

    // probes to evaluate.
    Set<StatusProbeEnum> probes = EnumSet.of(
        StatusProbeEnum.List, StatusProbeEnum.Head);


    // the PUT is conditional if requested, or if one of the
    // this is a performance creation, overwrite has not been requested,
    // this is not and etag write *and* conditional creation is available.
    // write is NOT conditional etag write.
    boolean conditionalPut = cCreate
        || !(overwrite || cEtag) && ccAvailable && createPerf;

    // skip the HEAD check for many reasons
    // old: the path is magic, it's an overwrite or the "create" performance is set.
    // new: also skip if any conditional create operation is in progress

View on GitHub (pinned to 2add963021)

Solutions

1. Set fs.s3a.create.conditional.enabled=true (the default) in the cluster or job configuration if the store supports conditional PUTs
2. Probe capability first: fs.hasPathCapability(path, "fs.s3a.create.conditional.enabled") and fall back to a plain create with overwrite=false when it returns false
3. For pure no-clobber semantics on stores without conditional writes, pre-check existence and use a unique destination name

Example fix

// before: assumes conditional writes exist on this FS
FSDataOutputStreamBuilder b = fs.createFile(dst);
b.opt("fs.option.create.conditional.overwrite", true);

// after
if (fs.hasPathCapability(dst, "fs.s3a.create.conditional.enabled")) {
  FSDataOutputStreamBuilder b = fs.createFile(dst);
  b.opt("fs.option.create.conditional.overwrite", true);
  out = b.build();
} else {
  out = fs.create(dst, false); // probe-based no-overwrite
}

// Probe the store capability before requesting conditional semantics
boolean cc = fs.hasPathCapability(path, "fs.s3a.create.conditional.enabled");
FSDataOutputStreamBuilder b = fs.createFile(path);
if (cc) {
  b.opt("fs.option.create.conditional.overwrite", true);
} else {
  // fall back to probe-based create
  out = fs.create(path, false);
}

Catch PathIOException from the create call and test whether the message is 'Conditional Writes Unavailable'; treat as configuration drift (enable fs.s3a.create.conditional.enabled or downgrade the create mode) - never as a transient error.

When it happens

Trigger: Using createFile(...) with the conditional-overwrite (or etag) create option on a cluster whose core-site.xml sets fs.s3a.create.conditional.enabled=false; frameworks or committers that request conditional writes running against such a configuration.

Common situations: Conditional create disabled for compatibility with stores lacking If-None-Match support (older MinIO/Ceph/S3 clones) while application code assumes the default (true); rolling upgrades where new code meets old cluster config; enforcing overwrite=false atomically on S3.

Related errors

• Unsupported block buffer "{name}"
• hasPathCapability on path `<p>' is not within a mount point
• This committer does not work with the filesystem of type {sc
• Bucket does not exist. Accessing with fs.s3a.endpoint set t
• Can't create root path

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3d5ca33fd0db4729. Report an issue: GitHub.