apache/iceberg · error · RuntimeException

Could not list sub directories, reached maximum depth: ${MAX

Error message

Could not list sub directories, reached maximum depth: ${MAX_EXECUTOR_LISTING_DEPTH}

What it means

During executor-side parallel listing in DeleteOrphanFiles, if subdirectories still remain after BFS traversal to MAX_EXECUTOR_LISTING_DEPTH (default 3), the listing task fails rather than silently ignoring deeper directories.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:518

      List<String> subDirs = Lists.newArrayList();
      List<String> files = Lists.newArrayList();

      Predicate<FileStatus> predicate = file -> file.getModificationTime() < olderThanTimestamp;

      while (dirs.hasNext()) {
        FileSystemWalker.listDirRecursivelyWithHadoop(
            dirs.next(),
            specs,
            predicate,
            hadoopConf.value().value(),
            MAX_EXECUTOR_LISTING_DEPTH,
            MAX_EXECUTOR_LISTING_DIRECT_SUB_DIRS,
            subDirs::add,
            files::add);
      }

      if (!subDirs.isEmpty()) {
        throw new RuntimeException(
            "Could not list sub directories, reached maximum depth: " + MAX_EXECUTOR_LISTING_DEPTH);
      }

      return files.iterator();
    }
  }

  private static class FindOrphanFiles
      implements MapPartitionsFunction<Tuple2<FileURI, FileURI>, String> {

    private final PrefixMismatchMode mode;
    private final SetAccumulator<Pair<String, String>> conflicts;

    FindOrphanFiles(PrefixMismatchMode mode, SetAccumulator<Pair<String, String>> conflicts) {
      this.mode = mode;
      this.conflicts = conflicts;
    }

View on GitHub (pinned to 86d9c8fc54)

Solutions

Increase spark.sql.catalog.<name>.spark.delete-orphan-files.max-executor-listing-depth (delete.orphan.files.max-executor-listing-depth property) above the depth of your tree
Increase MAX_EXECUTOR_LISTING_DIRECT_SUB_DIRS (max-direct-sub-dirs) if many sibling dirs cause deferred depth, or list shallower roots
Disable parallel listing so driver-side recursive listing (which has no depth cap) is used
Restructure/flatten the location hierarchy if practical

Example fix

// before (default depth 3 fails)
spark.sql("CALL cat.system.remove_orphan_files(table => 'db.tbl')");
// after
spark.conf.set("spark.sql.catalog.spark_catalog.delete-orphan-files.max-executor-listing-depth", "10");
spark.sql("CALL cat.system.remove_orphan_files(table => 'db.tbl')");

Defensive patterns

Strategy: validation

Validate before calling

long depth = countDirectoryDepth(tableLocationRoot);
if (depth > maxListingDepth) { raiseConfiguredDepthBeforeRunning(); }

Prevention

Set delete-orphan-files.max-executor-listing-depth above your tree depth
Avoid deeply nested hive-style partitions under the listing root
Use driver-side listing for very deep trees

When it happens

Trigger: Running deleteOrphanFiles on a table/location whose directory tree is deeper than MAX_EXECUTOR_LISTING_DEPTH under the listing root (parallel listing enabled via delete-orphan-files.parallelism style config).

Common situations: Very deep warehouse directory hierarchies (e.g. hive-style db.db/table/part=x/... nested many levels) combined with listing more than the max direct subdirectories per level (1024).

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/f3236174834ad3db. Report an issue: GitHub.