apache/druid · error · IOException

Read 0 bytes from segmentDir[%s]

Error message

Read 0 bytes from segmentDir[%s]

What it means

LocalIntermediaryDataManager.addSegment zips a task's segment directory into the intermediary shuffle storage. CompressionUtils.zip returning 0 bytes means the segment directory was empty or unreadable, so no usable segment artifact exists; it throws IOE rather than storing an empty segment file.

Source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/worker/shuffle/LocalIntermediaryDataManager.java:311

    if (!(segment.getShardSpec() instanceof BucketNumberedShardSpec)) {
      throw new IAE(
          "Invalid shardSpec type. Expected [%s] but got [%s]",
          BucketNumberedShardSpec.class.getName(),
          segment.getShardSpec().getClass().getName()
      );
    }
    final BucketNumberedShardSpec<?> bucketNumberedShardSpec = (BucketNumberedShardSpec<?>) segment.getShardSpec();

    //noinspection unused
    try (final Closer ignoredCloser = closer) {
      FileUtils.mkdirp(taskTempDir);

      // Temporary compressed file. Will be removed when taskTempDir is deleted.
      final File tempZippedFile = new File(taskTempDir, segment.getId().toString());
      final long unzippedSizeBytes = CompressionUtils.zip(segmentDir, tempZippedFile);
      if (unzippedSizeBytes == 0) {
        throw new IOE(
            "Read 0 bytes from segmentDir[%s]",
            segmentDir.getAbsolutePath()
        );
      }

      // Try copying the zipped segment to one of storage locations
      for (int i = 0; i < shuffleDataLocations.size(); i++) {
        final StorageLocation location = shuffleDataLocations.get(
            Math.floorMod(cursor.getAndIncrement(), shuffleDataLocations.size())
        );
        final String partitionFilePath = getPartitionFilePath(
            supervisorTaskId,
            subTaskId,
            segment.getInterval(),
            bucketNumberedShardSpec.getBucketId() // we must use the bucket ID instead of partition ID
        );
        File destFile = null;
        if (location.reserve(new FileNameCacheEntry(partitionFilePath, tempZippedFile.length()))) {

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Verify the task actually wrote the segment files into segmentDir before reporting the segment
2. Re-run the failing task to regenerate the segment
3. Check for concurrent cleanup (cleanup thresholds, disk reaper) racing with segment publication
4. Inspect filesystem permissions/health for the task's temp directory

Example fix

null

final long bytes = FileUtils.sizeOfDirectory(segmentDir);
if (bytes == 0) { throw new IllegalStateException("segmentDir " + segmentDir + " is empty before zip"); }

try { dataManager.addSegment(...); } catch (IOException e) { if (e.getMessage().startsWith("Read 0 bytes")) { rescheduleTask(); } else { throw e; } }

When it happens

Trigger: addSegment() called with a segmentDir whose contents total zero unzipped bytes — e.g. the directory is empty, files were already cleaned up, or reading the directory fails silently.

Common situations: Task directory cleaned concurrently (as in the isCleanedUpAfter3s test that exercises this), a task producing no rows for a segment, or disk/filesystem issues yielding an empty dir.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• Can't find location to handle segment[%s]
• Input stream is null
• Lock interval[%s] != task interval[%s]
• DynamicPartitionsSpec must be used for best-effort rollup
• Failed to add a row with timestamp[%s]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/25d633c8b63afa41. Report an issue: GitHub.