{"record":{"id":"27c176fb58fc8301","repo":"apache/druid","slug":"error-loading-s","errorCode":null,"errorMessage":"Error loading [%s]","messagePattern":"Error loading \\[(.+?)\\]","errorType":"exception","errorClass":"SegmentLoadingException","httpStatus":null,"severity":"error","filePath":"extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentPuller.java","lineNumber":293,"sourceCode":"                return getInputStream(path);\n              }\n            },\n            outFile\n        );\n\n        log.info(\n            \"Gunzipped %d bytes from [%s] to [%s]\",\n            result.size(),\n            path.toString(),\n            outFile.getAbsolutePath()\n        );\n        return result;\n      } else {\n        throw new SegmentLoadingException(\"Do not know how to handle file type at [%s]\", path.toString());\n      }\n    }\n    catch (IOException e) {\n      throw new SegmentLoadingException(e, \"Error loading [%s]\", path.toString());\n    }\n  }\n\n  private void emitMetrics(CompressionUtils.Format format, long size, long duration)\n  {\n    if (emitter == null) {\n      return;\n    }\n    ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();\n    metricBuilder.setDimension(\"format\", format);\n    emitter.emit(metricBuilder.setMetric(\"hdfs/pull/size\", size));\n    emitter.emit(metricBuilder.setMetric(\"hdfs/pull/duration\", duration));\n  }\n\n  public InputStream getInputStream(Path path) throws IOException\n  {\n    return buildFileObject(path.toUri(), config).openInputStream();\n  }","sourceCodeStart":275,"sourceCodeEnd":311,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentPuller.java#L275-L311","documentation":"The outer catch of getSegmentFiles converts any IOException encountered while reading from HDFS (fs.getFileStatus, fs.listFiles, fs.open, NativeIO.chunkedCopy, decompression streams) into SegmentLoadingException(e, \"Error loading [%s]\", path). The message is just 'Error loading <path>'; the actionable detail is in the wrapped IOException — typically a lost datanode, block missing, connection reset, or a corrupt/incomplete archive that surfaces as EOF/truncated-stream IOException.","triggerScenarios":"Any IOException during the pull: NameNode/datanode connectivity failure while opening or copying segment files, block-under-replication or corrupt block reads, decompression hitting a truncated .zip/.gz stream (stream reads throw IOException inside format.decompressDirectory/gunzip).","commonSituations":"HDFS cluster degradation during historical segment load; segments truncated by an aborted push; network firewall dropping long-lived datanode streams; corrupt archive left after a kill task raced with a load.","solutions":["Inspect getCause() (the IOException) to find the concrete failure: connect timeout, BlockMissingException, EOFException, etc.","For transient network/datanode issues, retry the load — callers can rely on the puller's RETRY_PREDICATE semantics and re-trigger segment loading.","If the cause is EOF/truncated stream, re-push or re-ingest the segment: the deep-storage artifact is corrupt or incomplete.","Check datanode health and replication (`hdfs fsck <path> -files -blocks`) and repair under-replicated/corrupt blocks.","Verify network reachability and dfs.client timeouts between the Druid host and the HDFS cluster."],"exampleFix":"// before: truncated index.zip in deep storage -> EOFException wrapped in 'Error loading'\n// $ hdfs dfs -get segment.zip && unzip -t segment.zip  -> unexpected end of file\n// after: re-push the segment\n// curl -X POST 'http://historical:8084/druid/historical/v1/loadSegment?...'\n// or re-run ingestion for the affected interval to rewrite index.zip","handlingStrategy":"retry","validationCode":"Path p = new Path(loadSpecPath);\nFileSystem fs = p.getFileSystem(config);\nif (!fs.exists(p) || fs.getFileStatus(p).getLen() == 0) { flagSegmentCorrupt(segmentId); }","typeGuard":null,"tryCatchPattern":"int attempts = 0;\nwhile (attempts++ < 3) {\n  try { return puller.getSegmentFiles(path, outDir); }\n  catch (SegmentLoadingException e) {\n    if (e.getCause() instanceof IOException && attempts < 3) { backoff(); continue; }\n    throw e;\n  }\n}","preventionTips":["Monitor HDFS cluster health and datanode connectivity from Druid nodes.","Verify pushers complete uploads atomically so truncated archives never land in deep storage.","Run `hdfs fsck` on failing paths to detect corrupt/under-replicated blocks.","Configure sensible dfs.client socket timeouts and retry policies in hdfs-site.xml."],"tags":["hdfs","io-exception","segment-loading","network"],"backgroundTag":"file-read-failed","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}