{"record":{"id":"7d2bcc6c60b2f2da","repo":"apache/druid","slug":"no-files-found-at-s","errorCode":null,"errorMessage":"No files found at [%s]","messagePattern":"No files found at \\[(.+?)\\]","errorType":"exception","errorClass":"SegmentLoadingException","httpStatus":null,"severity":"error","filePath":"extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentPuller.java","lineNumber":217,"sourceCode":"\n  FileUtils.FileCopyResult getSegmentFiles(final Path path, final File outDir) throws SegmentLoadingException\n  {\n    try {\n      FileUtils.mkdirp(outDir);\n    }\n    catch (IOException e) {\n      throw new SegmentLoadingException(e, \"\");\n    }\n    try {\n      final FileSystem fs = path.getFileSystem(config);\n      if (fs.getFileStatus(path).isDirectory()) {\n\n        // --------    directory     ---------\n        try {\n          return RetryUtils.retry(\n              () -> {\n                if (!fs.exists(path)) {\n                  throw new SegmentLoadingException(\"No files found at [%s]\", path.toString());\n                }\n\n                final RemoteIterator<LocatedFileStatus> children = fs.listFiles(path, false);\n                final FileUtils.FileCopyResult result = new FileUtils.FileCopyResult();\n                while (children.hasNext()) {\n                  final LocatedFileStatus child = children.next();\n                  final Path childPath = child.getPath();\n                  final String fname = childPath.getName();\n                  if (fs.getFileStatus(childPath).isDirectory()) {\n                    log.warn(\"[%s] is a child directory, skipping\", childPath.toString());\n                  } else {\n                    final File outFile = new File(outDir, fname);\n                    try (final FSDataInputStream in = fs.open(childPath)) {\n                      NativeIO.chunkedCopy(in, outFile);\n                    }\n                    result.addFile(outFile);\n                  }\n                }","sourceCodeStart":199,"sourceCodeEnd":235,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentPuller.java#L199-L235","documentation":"Inside the directory branch of getSegmentFiles, the retry lambda first checks fs.exists(path) and throws SegmentLoadingException(\"No files found at [%s]\") when the HDFS directory has disappeared. This runs within RetryUtils.retry, but SegmentLoadingException is not matched by the retry predicate (which only retries IOException/HdfsIOException), so the failure surfaces immediately. It means the segment directory expected in deep storage no longer exists.","triggerScenarios":"Loading a segment whose hdfs:// directory was concurrently deleted (kill task, killAll, retention policy, manual hdfs dfs -rm), or whose loadSpec path is wrong (typo in storageDirectory, moved/renamed deep storage location, cluster migration without copying segments).","commonSituations":"Historical trying to load a segment listed in metadata store but already removed from deep storage (metadata/deep-storage skew); HDFS name resolution after changing druid.storage.storageDirectory; lost data after NameNode restore; race between retention drop and query loading.","solutions":["Verify the path exists on HDFS: `hdfs dfs -ls <path from loadSpec>`; if absent, the segment is orphaned in the metadata store.","Run the metadata-store/deep-storage consistency cleanup (kill tasks or the Druid 'unused segments' cleanup) so orphaned segment rows are removed.","Check whether a concurrent kill/retention job deleted the segment and re-ingest the affected intervals if the data is still needed.","Confirm druid.storage.storageDirectory matches where segments were actually pushed (a moved deep storage makes all loadSpec paths stale).","If the deletion was accidental, restore from HDFS snapshots or backups and re-trigger segment loading."],"exampleFix":"// before: loadSpec points to /druid/segments/ds/2020_01_01... that was manually removed\n// $ hdfs dfs -ls hdfs://nn/druid/segments/ds/...  -> No such file or directory\n// after: remove orphan metadata rows so the segment is no longer served\n// curl -X DELETE 'http://coordinator:8081/druid/coordinator/v1/datasources/ds?kill=true&interval=2020-01-01/2020-01-02'\n// (or re-run ingestion to repopulate deep storage)","handlingStrategy":"validation","validationCode":"Path p = new Path(loadSpecPath);\nFileSystem fs = p.getFileSystem(config);\nif (!fs.exists(p)) {\n  markSegmentOrphan(segmentId); // remove from metadata store or re-ingest before loading\n}","typeGuard":null,"tryCatchPattern":"try { puller.getSegmentFiles(path, outDir); }\ncatch (SegmentLoadingException e) {\n  if (e.getMessage().startsWith(\"No files found\")) { dropOrphanSegment(segment); }\n}","preventionTips":["Run Druid's built-in metadata/deep-storage consistency cleanup regularly.","Never manually delete segment directories without also killing metadata rows.","Keep druid.storage.storageDirectory stable; migrating deep storage requires updating loadSpec paths.","Order operations so retention/kill tasks complete before nodes attempt to load affected segments."],"tags":["hdfs","segment-not-found","deep-storage","metadata-skew"],"backgroundTag":"file-not-found","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}