apache/druid · error · SegmentLoadingException

Error loading [%s]

Error message

Error loading [%s]

What it means

The outer catch of getSegmentFiles converts any IOException encountered while reading from HDFS (fs.getFileStatus, fs.listFiles, fs.open, NativeIO.chunkedCopy, decompression streams) into SegmentLoadingException(e, "Error loading [%s]", path). The message is just 'Error loading <path>'; the actionable detail is in the wrapped IOException — typically a lost datanode, block missing, connection reset, or a corrupt/incomplete archive that surfaces as EOF/truncated-stream IOException.

Source

Thrown at extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentPuller.java:293

                return getInputStream(path);
              }
            },
            outFile
        );

        log.info(
            "Gunzipped %d bytes from [%s] to [%s]",
            result.size(),
            path.toString(),
            outFile.getAbsolutePath()
        );
        return result;
      } else {
        throw new SegmentLoadingException("Do not know how to handle file type at [%s]", path.toString());
      }
    }
    catch (IOException e) {
      throw new SegmentLoadingException(e, "Error loading [%s]", path.toString());
    }
  }

  private void emitMetrics(CompressionUtils.Format format, long size, long duration)
  {
    if (emitter == null) {
      return;
    }
    ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
    metricBuilder.setDimension("format", format);
    emitter.emit(metricBuilder.setMetric("hdfs/pull/size", size));
    emitter.emit(metricBuilder.setMetric("hdfs/pull/duration", duration));
  }

  public InputStream getInputStream(Path path) throws IOException
  {
    return buildFileObject(path.toUri(), config).openInputStream();
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Inspect getCause() (the IOException) to find the concrete failure: connect timeout, BlockMissingException, EOFException, etc.
2. For transient network/datanode issues, retry the load — callers can rely on the puller's RETRY_PREDICATE semantics and re-trigger segment loading.
3. If the cause is EOF/truncated stream, re-push or re-ingest the segment: the deep-storage artifact is corrupt or incomplete.
4. Check datanode health and replication (`hdfs fsck <path> -files -blocks`) and repair under-replicated/corrupt blocks.
5. Verify network reachability and dfs.client timeouts between the Druid host and the HDFS cluster.

Example fix

// before: truncated index.zip in deep storage -> EOFException wrapped in 'Error loading'
// $ hdfs dfs -get segment.zip && unzip -t segment.zip  -> unexpected end of file
// after: re-push the segment
// curl -X POST 'http://historical:8084/druid/historical/v1/loadSegment?...'
// or re-run ingestion for the affected interval to rewrite index.zip

Path p = new Path(loadSpecPath);
FileSystem fs = p.getFileSystem(config);
if (!fs.exists(p) || fs.getFileStatus(p).getLen() == 0) { flagSegmentCorrupt(segmentId); }

int attempts = 0;
while (attempts++ < 3) {
  try { return puller.getSegmentFiles(path, outDir); }
  catch (SegmentLoadingException e) {
    if (e.getCause() instanceof IOException && attempts < 3) { backoff(); continue; }
    throw e;
  }
}

When it happens

Trigger: Any IOException during the pull: NameNode/datanode connectivity failure while opening or copying segment files, block-under-replication or corrupt block reads, decompression hitting a truncated .zip/.gz stream (stream reads throw IOException inside format.decompressDirectory/gunzip).

Common situations: HDFS cluster degradation during historical segment load; segments truncated by an aborted push; network firewall dropping long-lived datanode streams; corrupt archive left after a kill task raced with a load.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Unable to kill segment
• Do not know how to handle file type at [%s]
• %s
• %s
• Failed to fetch Avro schema id[%s] from registry. Check if t

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/27c176fb58fc8301. Report an issue: GitHub.