Failed to remove output directory [%s] for segment pulled fr

Error message

Failed to remove output directory [%s] for segment pulled from [%s]

What it means

GoogleDataSegmentPuller.getSegmentFiles() downloads a segment from GCS into a temp output directory. If any exception occurs during the pull, it attempts to clean up that directory with FileUtils.deleteDirectory; if that cleanup itself fails with IOException it logs this warning. The original pull error is still propagated to the caller; this message only signals leftover temp files.

Source

Thrown at extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleDataSegmentPuller.java:71

    try {
      FileUtils.mkdirp(outDir);

      final GoogleByteSource byteSource = new GoogleByteSource(storage, bucket, path);
      final FileUtils.FileCopyResult result = CompressionUtils.unzip(
          byteSource,
          outDir,
          GoogleUtils::isRetryable,
          false
      );
      LOG.info("Loaded %d bytes from [%s] to [%s]", result.size(), path, outDir.getAbsolutePath());
      return result;
    }
    catch (Exception e) {
      try {
        FileUtils.deleteDirectory(outDir);
      }
      catch (IOException ioe) {
        LOG.warn(
            ioe,
            "Failed to remove output directory [%s] for segment pulled from [%s]",
            outDir.getAbsolutePath(),
            path
        );
      }
      throw new SegmentLoadingException(e, e.getMessage());
    }
  }

  @Override
  public InputStream getInputStream(URI uri) throws IOException
  {
    String path = StringUtils.maybeRemoveLeadingSlash(uri.getPath());
    return storage.getInputStream(uri.getHost() != null ? uri.getHost() : uri.getAuthority(), path);
  }

  @Override

View on GitHub (pinned to 9b90983fd2)

Solutions

Fix the underlying pull failure first — check GCS permissions/bucket/object existence; this warning is secondary
Manually delete leftover temp directories (segment pulling dirs under the druid temp location) to reclaim disk space
Check filesystem permissions and disk health on the Druid historical/middleManager host
If recurring in tests, assert the original exception rather than relying on directory cleanup

Example fix

// before
catch (Exception e) {
  try { FileUtils.deleteDirectory(outDir); }
  catch (IOException ioe) { LOG.warn(ioe, "Failed to remove output directory [%s] ...", outDir.getAbsolutePath(), path); }
  throw e;
}
// after
catch (Exception e) {
  try {
    FileUtils.deleteDirectory(outDir);
  } catch (IOException ioe) {
    LOG.warn(ioe, "Failed to remove output directory [%s] for segment pulled from [%s]", outDir.getAbsolutePath(), path);
    outDir.deleteOnExit();
  }
  throw e;
}

Defensive patterns

Strategy: try-catch

Try / catch

try {
  puller.getSegmentFiles(path, outDir);
} catch (SegmentLoadingException e) {
  // handle the real pull failure; temp-dir cleanup warning is secondary
  LOG.error("Segment pull failed from [%s]: %s", path, e.getMessage());
}

Prevention

Monitor local disk health and permissions on Druid nodes
Ensure the Druid process user owns its temp directories
Alert on the underlying SegmentLoadingException, not this cleanup warning

When it happens

Trigger: A segment pull from GCS fails (e.g. network error, missing object, decompression failure) AND the temp output directory cannot be deleted (file lock, permission issue, disk problem on the local host).

Common situations: GCS outage or wrong permissions on the segment path combined with a local disk that won't allow deletion (read-only mount, running as non-root with root-owned temp dirs, NFS stale handles). Also common in tests that force pull failures while the outDir is not deletable.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/a9ba515d308ba637. Report an issue: GitHub.