{"record":{"id":"fa9853c483e793ab","repo":"apache/iceberg","slug":"failed-to-create-staging-directory-due-to-some-unk","errorCode":null,"errorMessage":"Failed to create staging directory due to some unknown reason: ","messagePattern":"Failed to create staging directory due to some unknown reason: ","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java","lineNumber":475,"sourceCode":"      throw new UncheckedIOException(e);\n    }\n  }\n\n  private void createStagingDirectoryIfNotExists() throws IOException, SecurityException {\n    if (!stagingDirectory.exists()) {\n      LOG.info(\n          \"Staging directory does not exist, trying to create one: {}\",\n          stagingDirectory.getAbsolutePath());\n      boolean createdStagingDirectory = stagingDirectory.mkdirs();\n      if (createdStagingDirectory) {\n        LOG.info(\"Successfully created staging directory: {}\", stagingDirectory.getAbsolutePath());\n      } else {\n        if (stagingDirectory.exists()) {\n          LOG.info(\n              \"Successfully created staging directory by another process: {}\",\n              stagingDirectory.getAbsolutePath());\n        } else {\n          throw new IOException(\n              \"Failed to create staging directory due to some unknown reason: \"\n                  + stagingDirectory.getAbsolutePath());\n        }\n      }\n    }\n  }\n\n  @SuppressWarnings({\"checkstyle:NoFinalizer\", \"Finalize\", \"deprecation\"})\n  @Override\n  protected void finalize() throws Throwable {\n    super.finalize();\n    if (!closed) {\n      close(false); // releasing resources is more important than printing the warning\n      String trace = Joiner.on(\"\\n\\t\").join(Arrays.copyOfRange(createStack, 1, createStack.length));\n      LOG.warn(\"Unclosed output stream created by:\\n\\t{}\", trace);\n    }\n  }\n","sourceCodeStart":457,"sourceCodeEnd":493,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java#L457-L493","documentation":"createStagingDirectoryIfNotExists() throws an IOException when mkdirs() failed and the directory still does not exist afterwards. It indicates the staging directory could not be created for an undiagnosed reason (permissions, disk, path is a file).","triggerScenarios":"File.mkdirs() returns false and stagingDirectory.exists() is still false when S3OutputStream opens its first part stream.","commonSituations":"Staging path's parent is a regular file; no write permission on parent; read-only container filesystem; NFS/EFS hang or quota issues.","solutions":["Verify the staging path's parent exists and is writable (and the path itself is not a file).","Pre-create the directory with correct ownership before starting the job.","Point s3.staging-directory at a guaranteed-writable local volume (e.g. /tmp or a mounted scratch disk).","Check filesystem quotas and mount status on container images."],"exampleFix":"// before\nprops.put(\"s3.staging-directory\", \"/read-only-fs/staging\");\n// after\nFiles.createDirectories(Paths.get(\"/tmp/iceberg-staging\"));\nprops.put(\"s3.staging-directory\", \"/tmp/iceberg-staging\");","handlingStrategy":"validation","validationCode":"// Java\nFile dir = new File(props.stagingDirectory());\nif (!dir.isDirectory() && !dir.mkdirs()) {\n  throw new IllegalStateException(\"Cannot create staging dir: \" + dir.getAbsolutePath());\n}","typeGuard":null,"tryCatchPattern":"// Java\ntry {\n  out.write(data);\n} catch (UncheckedIOException e) {\n  if (e.getCause().getMessage().contains(\"staging directory\")) {\n    LOG.error(\"Fix staging path: {}\", e.getCause().getMessage());\n  }\n  throw e;\n}","preventionTips":["Pre-create the staging directory with correct ownership before job start","Ensure the staging parent path is writable and not occupied by a regular file","Point staging at guaranteed-writable local scratch volumes"],"tags":["s3","staging","filesystem","mkdir"],"backgroundTag":"file-write-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}