{"record":{"id":"f3236174834ad3db","repo":"apache/iceberg","slug":"could-not-list-sub-directories-reached-maximum-de-f32361","errorCode":null,"errorMessage":"Could not list sub directories, reached maximum depth: ${MAX_EXECUTOR_LISTING_DEPTH}","messagePattern":"Could not list sub directories, reached maximum depth: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java","lineNumber":518,"sourceCode":"      List<String> subDirs = Lists.newArrayList();\n      List<String> files = Lists.newArrayList();\n\n      Predicate<FileStatus> predicate = file -> file.getModificationTime() < olderThanTimestamp;\n\n      while (dirs.hasNext()) {\n        FileSystemWalker.listDirRecursivelyWithHadoop(\n            dirs.next(),\n            specs,\n            predicate,\n            hadoopConf.value().value(),\n            MAX_EXECUTOR_LISTING_DEPTH,\n            MAX_EXECUTOR_LISTING_DIRECT_SUB_DIRS,\n            subDirs::add,\n            files::add);\n      }\n\n      if (!subDirs.isEmpty()) {\n        throw new RuntimeException(\n            \"Could not list sub directories, reached maximum depth: \" + MAX_EXECUTOR_LISTING_DEPTH);\n      }\n\n      return files.iterator();\n    }\n  }\n\n  private static class FindOrphanFiles\n      implements MapPartitionsFunction<Tuple2<FileURI, FileURI>, String> {\n\n    private final PrefixMismatchMode mode;\n    private final SetAccumulator<Pair<String, String>> conflicts;\n\n    FindOrphanFiles(PrefixMismatchMode mode, SetAccumulator<Pair<String, String>> conflicts) {\n      this.mode = mode;\n      this.conflicts = conflicts;\n    }\n","sourceCodeStart":500,"sourceCodeEnd":536,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java#L500-L536","documentation":"During executor-side parallel listing in DeleteOrphanFiles, if subdirectories still remain after BFS traversal to MAX_EXECUTOR_LISTING_DEPTH (default 3), the listing task fails rather than silently ignoring deeper directories.","triggerScenarios":"Running deleteOrphanFiles on a table/location whose directory tree is deeper than MAX_EXECUTOR_LISTING_DEPTH under the listing root (parallel listing enabled via delete-orphan-files.parallelism style config).","commonSituations":"Very deep warehouse directory hierarchies (e.g. hive-style db.db/table/part=x/... nested many levels) combined with listing more than the max direct subdirectories per level (1024).","solutions":["Increase spark.sql.catalog.<name>.spark.delete-orphan-files.max-executor-listing-depth (delete.orphan.files.max-executor-listing-depth property) above the depth of your tree","Increase MAX_EXECUTOR_LISTING_DIRECT_SUB_DIRS (max-direct-sub-dirs) if many sibling dirs cause deferred depth, or list shallower roots","Disable parallel listing so driver-side recursive listing (which has no depth cap) is used","Restructure/flatten the location hierarchy if practical"],"exampleFix":"// before (default depth 3 fails)\nspark.sql(\"CALL cat.system.remove_orphan_files(table => 'db.tbl')\");\n// after\nspark.conf.set(\"spark.sql.catalog.spark_catalog.delete-orphan-files.max-executor-listing-depth\", \"10\");\nspark.sql(\"CALL cat.system.remove_orphan_files(table => 'db.tbl')\");","handlingStrategy":"validation","validationCode":"long depth = countDirectoryDepth(tableLocationRoot);\nif (depth > maxListingDepth) { raiseConfiguredDepthBeforeRunning(); }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set delete-orphan-files.max-executor-listing-depth above your tree depth","Avoid deeply nested hive-style partitions under the listing root","Use driver-side listing for very deep trees"],"tags":["spark","delete-orphan-files","listing-depth","directory-depth"],"backgroundTag":"value-out-of-range","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}