{"record":{"id":"d5be54e484805e8b","repo":"apache/iceberg","slug":"could-not-list-sub-directories-reached-maximum-de-d5be54","errorCode":null,"errorMessage":"Could not list sub directories, reached maximum depth: {MAX_EXECUTOR_LISTING_DEPTH}","messagePattern":"Could not list sub directories, reached maximum depth: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java","lineNumber":518,"sourceCode":"      List<String> subDirs = Lists.newArrayList();\n      List<String> files = Lists.newArrayList();\n\n      Predicate<FileStatus> predicate = file -> file.getModificationTime() < olderThanTimestamp;\n\n      while (dirs.hasNext()) {\n        FileSystemWalker.listDirRecursivelyWithHadoop(\n            dirs.next(),\n            specs,\n            predicate,\n            hadoopConf.value().value(),\n            MAX_EXECUTOR_LISTING_DEPTH,\n            MAX_EXECUTOR_LISTING_DIRECT_SUB_DIRS,\n            subDirs::add,\n            files::add);\n      }\n\n      if (!subDirs.isEmpty()) {\n        throw new RuntimeException(\n            \"Could not list sub directories, reached maximum depth: \" + MAX_EXECUTOR_LISTING_DEPTH);\n      }\n\n      return files.iterator();\n    }\n  }\n\n  private static class FindOrphanFiles\n      implements MapPartitionsFunction<Tuple2<FileURI, FileURI>, String> {\n\n    private final PrefixMismatchMode mode;\n    private final SetAccumulator<Pair<String, String>> conflicts;\n\n    FindOrphanFiles(PrefixMismatchMode mode, SetAccumulator<Pair<String, String>> conflicts) {\n      this.mode = mode;\n      this.conflicts = conflicts;\n    }\n","sourceCodeStart":500,"sourceCodeEnd":536,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java#L500-L536","documentation":"DeleteOrphanFiles performs distributed parallel listing in Spark executors up to MAX_EXECUTOR_LISTING_DEPTH levels of subdirectories. If subdirectories remain unprocessed at that depth, the action fails rather than silently missing files, preventing incomplete orphan-file deletion decisions.","triggerScenarios":"Running DeleteOrphanFiles on a table whose directory tree under the location is deeper than MAX_EXECUTOR_LISTING_DEPTH, so the Listing iterator still has subDirs after traversing the allowed depth.","commonSituations":"Very deep partitioning schemes (e.g. year/month/day/hour/... plus app-level folders); pointing the action at a location containing unrelated nested directory trees.","solutions":["Increase spark.sql.catalog.<catalog>.spark.delete-orphan-files.max-listing-depth (TableProperties.DELETE_ORPHAN_FILES_MAX_LISTING_DEPTH) to a larger value","Point the action at a deeper subdirectory (e.g. the table's data folder) so fewer levels need traversal","Restructure the data location to a shallower partition layout"],"exampleFix":"// before\nspark.conf.unset(\"spark.sql.catalog.my_cat.spark.delete-orphan-files.max-listing-depth\");\n// after\nspark.conf.set(\"spark.sql.catalog.my_cat.spark.delete-orphan-files.max-listing-depth\", \"10\");","handlingStrategy":"validation","validationCode":"long depth = locationDepth(tableLocation); // compute max subdirectory depth\nint maxDepth = conf.getInt(\"spark.sql.catalog.\" + catalog + \".spark.delete-orphan-files.max-listing-depth\", 3);\nif (depth > maxDepth) { spark.conf.set(...max-listing-depth..., String.valueOf(depth + 1)); }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set delete-orphan-files.max-listing-depth above your deepest partition nesting","Point the action at the data subdirectory rather than a bucket root","Keep partition layouts shallow"],"tags":["spark","iceberg","listing-depth","configuration"],"backgroundTag":"value-out-of-range","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}