apache/iceberg · warning

Failed to delete staging file: {}

Error message

Failed to delete staging file: {}

What it means

S3OutputStream buffers data to local staging files, then uploads them as multipart parts asynchronously. After each part upload completes, the staging file is deleted; if that filesystem deletion throws IOException, it logs this warning with the file path and exception. Upload results are unaffected, but staging files can accumulate in the temp directory.

Source

Thrown at aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java:339

              UploadPartRequest uploadRequest = requestBuilder.build();

              CompletableFuture<CompletedPart> future =
                  CompletableFuture.supplyAsync(
                          () -> {
                            UploadPartResponse response =
                                s3.uploadPart(uploadRequest, RequestBody.fromFile(f));
                            return CompletedPart.builder()
                                .eTag(response.eTag())
                                .partNumber(uploadRequest.partNumber())
                                .build();
                          },
                          executorService)
                      .whenComplete(
                          (result, thrown) -> {
                            try {
                              Files.deleteIfExists(f.toPath());
                            } catch (IOException e) {
                              LOG.warn("Failed to delete staging file: {}", f, e);
                            }

                            if (thrown != null) {
                              // Exception observed here will be thrown as part of
                              // CompletionException
                              // when we will join completable futures.
                              LOG.error("Failed to upload part: {}", uploadRequest, thrown);
                            }
                          });

              multiPartMap.put(f, future);
            });
  }

  private void completeMultiPartUpload() {
    Preconditions.checkState(closed, "Complete upload called on open stream: " + location);

    List<CompletedPart> completedParts;

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Ensure the temp directory (java.io.tmpdir / s3.staging-directory) is writable and has free space
2. Manually clean leftover staging files from previous runs
3. Point the staging directory to a reliable local disk instead of NFS
4. Check the attached IOException for the exact OS-level reason

Example fix

// before
props.put("s3.staging-directory", "/mnt/nfs/tmp");
// after: local disk with adequate space
props.put("s3.staging-directory", "/var/tmp/iceberg-staging");

// before writing, verify staging dir is usable
File dir = new File(System.getProperty("java.io.tmpdir"));
File probe = File.createTempFile("iceberg-probe", null, dir); probe.deleteOnExit();

try (OutputStream out = outputFile.create()) { write(out); } // staging cleanup failures are warnings; check logs for leftover files

When it happens

Trigger: Writing an S3 output file where the temp staging directory's file cannot be deleted — read-only or full temp filesystem, file locked by another process, or permissions changed mid-run.

Common situations: Containers with small/full /tmp volumes; concurrent uploads on NFS-mounted temp dirs; jobs running under restricted users whose tmpdir permissions drift.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• An error occurred while aborting the stream
• Cannot initialize S3FileIOAwsClientFactory, missing no-arg c
• Cannot initialize S3FileIOAwsClientFactory, %s does not impl
• Failed to create S3 analytics accelerator input stream for:
• Interrupted when waiting for deletions to complete

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/592cc69fd968ffaf. Report an issue: GitHub.