apache/iceberg · error · IOException

Failed to create staging directory due to some unknown…

Error message

Failed to create staging directory due to some unknown reason: 

What it means

createStagingDirectoryIfNotExists() throws an IOException when mkdirs() failed and the directory still does not exist afterwards. It indicates the staging directory could not be created for an undiagnosed reason (permissions, disk, path is a file).

Solutions

1. Verify the staging path's parent exists and is writable (and the path itself is not a file).
2. Pre-create the directory with correct ownership before starting the job.
3. Point s3.staging-directory at a guaranteed-writable local volume (e.g. /tmp or a mounted scratch disk).
4. Check filesystem quotas and mount status on container images.

Example fix

// before
props.put("s3.staging-directory", "/read-only-fs/staging");
// after
Files.createDirectories(Paths.get("/tmp/iceberg-staging"));
props.put("s3.staging-directory", "/tmp/iceberg-staging");

// Java
File dir = new File(props.stagingDirectory());
if (!dir.isDirectory() && !dir.mkdirs()) {
  throw new IllegalStateException("Cannot create staging dir: " + dir.getAbsolutePath());
}

// Java
try {
  out.write(data);
} catch (UncheckedIOException e) {
  if (e.getCause().getMessage().contains("staging directory")) {
    LOG.error("Fix staging path: {}", e.getCause().getMessage());
  }
  throw e;
}

When it happens

Trigger: File.mkdirs() returns false and stagingDirectory.exists() is still false when S3OutputStream opens its first part stream.

Common situations: Staging path's parent is a regular file; no write permission on parent; read-only container filesystem; NFS/EFS hang or quota issues.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Failed to create output stream for location:
• Failed to delete staging file
• An error occurred while aborting the stream
• An error occurred while closing the stream
• Can't create file

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/fa9853c483e793ab. Report an issue: GitHub.

Appendix: source

Thrown at aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java:475

      throw new UncheckedIOException(e);
    }
  }

  private void createStagingDirectoryIfNotExists() throws IOException, SecurityException {
    if (!stagingDirectory.exists()) {
      LOG.info(
          "Staging directory does not exist, trying to create one: {}",
          stagingDirectory.getAbsolutePath());
      boolean createdStagingDirectory = stagingDirectory.mkdirs();
      if (createdStagingDirectory) {
        LOG.info("Successfully created staging directory: {}", stagingDirectory.getAbsolutePath());
      } else {
        if (stagingDirectory.exists()) {
          LOG.info(
              "Successfully created staging directory by another process: {}",
              stagingDirectory.getAbsolutePath());
        } else {
          throw new IOException(
              "Failed to create staging directory due to some unknown reason: "
                  + stagingDirectory.getAbsolutePath());
        }
      }
    }
  }

  @SuppressWarnings({"checkstyle:NoFinalizer", "Finalize", "deprecation"})
  @Override
  protected void finalize() throws Throwable {
    super.finalize();
    if (!closed) {
      close(false); // releasing resources is more important than printing the warning
      String trace = Joiner.on("\n\t").join(Arrays.copyOfRange(createStack, 1, createStack.length));
      LOG.warn("Unclosed output stream created by:\n\t{}", trace);
    }
  }

View on GitHub (pinned to 86d9c8fc54)