{"record":{"id":"71c28a845bf67123","repo":"apache/iceberg","slug":"could-not-list-sub-directories-reached-maximum-de-71c28a","errorCode":null,"errorMessage":"Could not list sub directories, reached maximum depth: ","messagePattern":"Could not list sub directories, reached maximum depth: ","errorType":"exception","errorClass":"java.lang.RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java","lineNumber":518,"sourceCode":"      List<String> subDirs = Lists.newArrayList();\n      List<String> files = Lists.newArrayList();\n\n      Predicate<FileStatus> predicate = file -> file.getModificationTime() < olderThanTimestamp;\n\n      while (dirs.hasNext()) {\n        FileSystemWalker.listDirRecursivelyWithHadoop(\n            dirs.next(),\n            specs,\n            predicate,\n            hadoopConf.value().value(),\n            MAX_EXECUTOR_LISTING_DEPTH,\n            MAX_EXECUTOR_LISTING_DIRECT_SUB_DIRS,\n            subDirs::add,\n            files::add);\n      }\n\n      if (!subDirs.isEmpty()) {\n        throw new RuntimeException(\n            \"Could not list sub directories, reached maximum depth: \" + MAX_EXECUTOR_LISTING_DEPTH);\n      }\n\n      return files.iterator();\n    }\n  }\n\n  private static class FindOrphanFiles\n      implements MapPartitionsFunction<Tuple2<FileURI, FileURI>, String> {\n\n    private final PrefixMismatchMode mode;\n    private final SetAccumulator<Pair<String, String>> conflicts;\n\n    FindOrphanFiles(PrefixMismatchMode mode, SetAccumulator<Pair<String, String>> conflicts) {\n      this.mode = mode;\n      this.conflicts = conflicts;\n    }\n","sourceCodeStart":500,"sourceCodeEnd":536,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java#L500-L536","documentation":"The executor-side listing task in DeleteOrphanFiles walks directories up to spark.sql.files.max... / MAX_EXECUTOR_LISTING_DEPTH levels. If subdirectories still remain at the maximum depth, it aborts with this RuntimeException rather than silently skipping files, preventing incomplete orphan detection.","triggerScenarios":"Running remove_orphan_files on a table location whose directory tree is deeper than MAX_EXECUTOR_LISTING_DEPTH (default 3) beneath the listed root, and the parallel listing still finds subDirs at that depth.","commonSituations":"Very deep partitioning layouts (e.g. year/month/day/hour/...) or nested path conventions under the table location; partition columns beyond the depth limit.","solutions":["Increase the depth via the procedure's maxDepth / listing depth parameter (older_file... no; use max_depth option or the internal maxListingDepth override)","Point the action at a shallower location (e.g. a specific partition prefix) and run in batches","Restructure the data layout if deep nesting is accidental"],"exampleFix":"// before\nCALL iceberg.system.remove_orphan_files(table => 'db.t') -- deeper than default depth\n// after\nCALL iceberg.system.remove_orphan_files(table => 'db.t', max_depth => 10);","handlingStrategy":"fallback","validationCode":"// Estimate layout depth under the table location before listing\n// hadoop fs -count on the location; ensure partition nesting < max listing depth","typeGuard":null,"tryCatchPattern":"try { removeOrphanFiles(); } catch (RuntimeException e) { if (e.getMessage().startsWith(\"Could not list sub directories\")) { /* increase depth or target a shallower prefix */ } }","preventionTips":["Set the max listing depth parameter explicitly for deep partition layouts","Avoid unnecessary directory nesting in table locations","Run the action on narrower prefixes (partition subtrees) when trees are deep"],"tags":["spark","orphan-files","directory-depth","filesystem"],"backgroundTag":"value-out-of-range","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}