Failed to remove output directory [%s] for segment pulled fr

Error message

Failed to remove output directory [%s] for segment pulled from [%s]

What it means

AzureDataSegmentPuller.getSegmentFiles() downloads a segment zip from Azure blob storage and extracts it into outDir. If extraction/read throws IOException, it attempts to delete the partially populated output directory; this warning is logged when that cleanup delete itself fails with a second IOException. The original IOException is still rethrown to the caller.

Source

Thrown at extensions-core/azure-extensions/src/main/java/org/apache/druid/storage/azure/AzureDataSegmentPuller.java:89

      final String actualBlobPath = AzureUtils.maybeRemoveAzurePathPrefix(blobPath, azureAccountConfig.getBlobStorageEndpoint());

      final ByteSource byteSource = byteSourceFactory.create(containerName, actualBlobPath, azureStorage);
      final FileUtils.FileCopyResult result = CompressionUtils.unzip(
          byteSource,
          outDir,
          AzureUtils.AZURE_RETRY,
          false
      );

      log.info("Loaded %d bytes from [%s] to [%s]", result.size(), actualBlobPath, outDir.getAbsolutePath());
      return result;
    }
    catch (IOException e) {
      try {
        FileUtils.deleteDirectory(outDir);
      }
      catch (IOException ioe) {
        log.warn(
            ioe,
            "Failed to remove output directory [%s] for segment pulled from [%s]",
            outDir.getAbsolutePath(),
            blobPath
        );
      }
      throw new SegmentLoadingException(e, e.getMessage());
    }
  }
}

View on GitHub (pinned to 9b90983fd2)

Solutions

Fix the underlying pull failure first: re-upload or re-download the corrupted segment blob
Check disk space and permissions on druid.segmentCache location
Manually delete the leftover output directory and retry the segment load
Inspect the original IOException (rethrown) for the root cause; the warning is secondary cleanup noise
Restart the process if files are locked and the directory cannot be removed

Example fix

// before: retrying with a corrupt blob present
// after: delete the corrupted Azure blob and re-push the segment
bin/druid segmenttool push --file=segment.zip --bucket=druid --path=wikipedia/2019-04-02T00:00:00.000Z_2019-04-03T00:00:00.000Z/...

Defensive patterns

Strategy: retry

Validate before calling

// check disk space and permissions on the segment cache dir before pulling
if (Files.getFileStore(cacheDir).getUsableSpace() < requiredBytes) {
    throw new IllegalStateException("Insufficient disk space for segment cache");
}

Try / catch

try {
  puller.getSegmentFiles(descriptor, outDir);
} catch (SegmentLoadingException e) {
  FileUtils.deleteQuietly(outDir); // clean up what the puller could not
  retryOrRedownload(e);
}

Prevention

Verify segment blobs are intact after upload (checksums)
Keep druid.segmentCache on a disk with ample free space
Ensure process owns write permissions on the cache directory
Re-push segments that repeatedly fail extraction

When it happens

Trigger: Segment pull fails mid-extract (corrupt zip blob, truncated download, disk I/O error) AND the subsequent FileUtils.deleteDirectory(outDir) cannot remove the directory (locked files, permissions, full disk).

Common situations: Corrupted or partially uploaded segment blob in Azure storage; local disk full on the Druid historical; permission issues on the segment cache directory; files held open by another process.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/efcfd7fe7fdaa47a. Report an issue: GitHub.