{"record":{"id":"2533fd80e3d4c3ad","repo":"apache/iceberg","slug":"could-not-list-sub-directories-reached-maximum-de","errorCode":null,"errorMessage":"Could not list sub directories, reached maximum depth: ","messagePattern":"Could not list sub directories, reached maximum depth: ","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java","lineNumber":518,"sourceCode":"      List<String> subDirs = Lists.newArrayList();\n      List<String> files = Lists.newArrayList();\n\n      Predicate<FileStatus> predicate = file -> file.getModificationTime() < olderThanTimestamp;\n\n      while (dirs.hasNext()) {\n        FileSystemWalker.listDirRecursivelyWithHadoop(\n            dirs.next(),\n            specs,\n            predicate,\n            hadoopConf.value().value(),\n            MAX_EXECUTOR_LISTING_DEPTH,\n            MAX_EXECUTOR_LISTING_DIRECT_SUB_DIRS,\n            subDirs::add,\n            files::add);\n      }\n\n      if (!subDirs.isEmpty()) {\n        throw new RuntimeException(\n            \"Could not list sub directories, reached maximum depth: \" + MAX_EXECUTOR_LISTING_DEPTH);\n      }\n\n      return files.iterator();\n    }\n  }\n\n  private static class FindOrphanFiles\n      implements MapPartitionsFunction<Tuple2<FileURI, FileURI>, String> {\n\n    private final PrefixMismatchMode mode;\n    private final SetAccumulator<Pair<String, String>> conflicts;\n\n    FindOrphanFiles(PrefixMismatchMode mode, SetAccumulator<Pair<String, String>> conflicts) {\n      this.mode = mode;\n      this.conflicts = conflicts;\n    }\n","sourceCodeStart":500,"sourceCodeEnd":536,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java#L500-L536","documentation":"The distributed listing task in DeleteOrphanFilesSparkAction recurses into subdirectories up to MAX_EXECUTOR_LISTING_DEPTH (2000). If after the maximum depth there are still unexplored subdirectories, it throws this RuntimeException instead of silently returning an incomplete orphan-file list, because a truncated listing could cause live files to be judged orphan and deleted.","triggerScenarios":"Running deleteOrphanFiles on a location with directory nesting deeper than 2000 levels (or a listing that expands beyond the depth cap), e.g. deeply hierarchical data layouts or pathological directory structures in the table location.","commonSituations":"Object-store buckets with extremely deep prefixes (each '/' counted), corrupted/malicious directory trees, or migrations from systems that created very deep folder hierarchies.","solutions":["Flatten the physical directory structure so nesting is far below the 2000-level cap.","If the tree is legitimately deep, list in stages: run the action per sub-prefix (olderLocation/location) and merge results manually.","Investigate the directory tree for accidental runaway nesting (e.g. a job writing timestamps-as-nested-dirs) and clean it up with non-destructive tooling first.","Increase MAX_EXECUTOR_LISTING_DEPTH in a patched build only if you fully understand the orphan-detection risk of deeper listings."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Before running, sanity-check nesting depth of the table location\nlong maxDepth = java.nio.file.Files.walk(dir).filter(Files::isDirectory)\n    .mapToLong(p -> p.getNameCount()).max().orElse(0);\nif (maxDepth > 2000) throw new IllegalStateException(\"Listing depth \" + maxDepth + \" exceeds executor cap\");","typeGuard":null,"tryCatchPattern":"try {\n  action.execute();\n} catch (RuntimeException e) {\n  if (e.getMessage().startsWith(\"Could not list sub directories, reached maximum depth\")) {\n    throw new IllegalStateException(\"Table location too deeply nested; flatten before orphan cleanup\", e);\n  } else throw e;\n}","preventionTips":["Avoid generating data locations with timestamp-per-directory hierarchies that nest unboundedly.","Periodically audit table location depth with a dry listing before orphan-file deletion.","Run orphan-file cleanup scoped to shallow prefixes rather than one giant deep root."],"tags":["spark","delete-orphan-files","listing","depth-limit"],"backgroundTag":"value-out-of-range","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}