apache/hadoop · error · IOException

Failed to delete temporary files while closing stream: '%s'

Error message

Failed to delete temporary files while closing stream: '%s'

What it means

GoogleHadoopOutputStream buffers large writes into temporary GCS objects and deletes them asynchronously; close() waits on those deletion futures (tmpDeletionFutures). If any future fails (ExecutionException) or the closing thread is interrupted, close() throws IOException wrapping the cause. Note the data in the destination object is already committed - only temp cleanup failed.

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleHadoopOutputStream.java:267

    commitTempFile();

    try {
      tmpOut.close();
    } finally {
      tmpOut = null;
    }
    tmpGcsPath = null;
    tmpIndex = -1;

    LOG.trace("close(): Awaiting {} deletionFutures", tmpDeletionFutures.size());
    for (Future<?> deletion : tmpDeletionFutures) {
      try {
        deletion.get();
      } catch (ExecutionException | InterruptedException e) {
        if (e instanceof InterruptedException) {
          Thread.currentThread().interrupt();
        }
        throw new IOException(
            String.format(
                "Failed to delete temporary files while closing stream: '%s'", dstGcsPath),
            e);
      }
    }
  }

  private void throwIfNotOpen() throws IOException {
    if (tmpOut == null) {
      throw new ClosedChannelException();
    }
  }

  @Override
  public boolean hasCapability(String capability) {
    checkArgument(!isNullOrEmpty(capability), "capability must not be null or empty string");
    switch (Ascii.toLowerCase(capability)) {
    case StreamCapabilities.HFLUSH:

View on GitHub (pinned to 2add963021)

Solutions

Inspect the wrapped cause (e.getCause()) to identify the real failure - permission, network, or interrupt.
Verify the service account can delete objects under the configured temporary directory; clean stale temp objects (hadoop- prefixed leftovers) after failures.
Avoid interrupting writer threads during close; ensure the connector's executor is not externally shut down.
Treat this as a cleanup warning for the committed output: data is written, but investigate why deletion failed.

Example fix

// before
try (FSDataOutputStream out = fs.create(path)) {
  out.write(data);
} // close() may throw 'Failed to delete temporary files while closing stream'

// after
try (FSDataOutputStream out = fs.create(path)) {
  out.write(data);
} catch (IOException e) {
  // output is committed; diagnose cleanup failure via e.getCause()
  LOG.warn("temp cleanup failed for {}", path, e.getCause());
  // optionally sweep the configured temp dir for leftovers
}

Defensive patterns

Strategy: try-catch

Validate before calling

null

Try / catch

try (FSDataOutputStream out = fs.create(path)) {
  out.write(data);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Failed to delete temporary files")) {
    // committed output is intact; inspect e.getCause() (perm/network/interrupt) and sweep leftover temp objects
  } else {
    throw e;
  }
}

Prevention

Grant the writer identity storage.objects.delete on the configured temp location.
Never interrupt writer threads during close(); let shutdown hooks drain gracefully.
After this failure, sweep the connector's temp directory for stale objects to avoid unbounded accumulation.

When it happens

Trigger: out.close() while a background deletion fails: GCS API/network errors, service account lacking storage.objects.delete on the temp location, or executor shutdown; thread interruption during close (the code restores the interrupt flag before throwing).

Common situations: Aggressive shutdown hooks interrupting writer threads; credentials expiring or permissions changed during very long writes; leftover gs://<bucket>/<tmp-dir> objects accumulating after this failure; executors shared with code that shuts them down early.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/26a7137f6576e8db. Report an issue: GitHub.