apache/iceberg · error · RuntimeException

Could not list sub directories, reached maximum depth: {MAX_

Error message

Could not list sub directories, reached maximum depth: {MAX_EXECUTOR_LISTING_DEPTH}

What it means

DeleteOrphanFiles performs distributed parallel listing in Spark executors up to MAX_EXECUTOR_LISTING_DEPTH levels of subdirectories. If subdirectories remain unprocessed at that depth, the action fails rather than silently missing files, preventing incomplete orphan-file deletion decisions.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:518

      List<String> subDirs = Lists.newArrayList();
      List<String> files = Lists.newArrayList();

      Predicate<FileStatus> predicate = file -> file.getModificationTime() < olderThanTimestamp;

      while (dirs.hasNext()) {
        FileSystemWalker.listDirRecursivelyWithHadoop(
            dirs.next(),
            specs,
            predicate,
            hadoopConf.value().value(),
            MAX_EXECUTOR_LISTING_DEPTH,
            MAX_EXECUTOR_LISTING_DIRECT_SUB_DIRS,
            subDirs::add,
            files::add);
      }

      if (!subDirs.isEmpty()) {
        throw new RuntimeException(
            "Could not list sub directories, reached maximum depth: " + MAX_EXECUTOR_LISTING_DEPTH);
      }

      return files.iterator();
    }
  }

  private static class FindOrphanFiles
      implements MapPartitionsFunction<Tuple2<FileURI, FileURI>, String> {

    private final PrefixMismatchMode mode;
    private final SetAccumulator<Pair<String, String>> conflicts;

    FindOrphanFiles(PrefixMismatchMode mode, SetAccumulator<Pair<String, String>> conflicts) {
      this.mode = mode;
      this.conflicts = conflicts;
    }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Increase spark.sql.catalog.<catalog>.spark.delete-orphan-files.max-listing-depth (TableProperties.DELETE_ORPHAN_FILES_MAX_LISTING_DEPTH) to a larger value
2. Point the action at a deeper subdirectory (e.g. the table's data folder) so fewer levels need traversal
3. Restructure the data location to a shallower partition layout

Example fix

// before
spark.conf.unset("spark.sql.catalog.my_cat.spark.delete-orphan-files.max-listing-depth");
// after
spark.conf.set("spark.sql.catalog.my_cat.spark.delete-orphan-files.max-listing-depth", "10");

long depth = locationDepth(tableLocation); // compute max subdirectory depth
int maxDepth = conf.getInt("spark.sql.catalog." + catalog + ".spark.delete-orphan-files.max-listing-depth", 3);
if (depth > maxDepth) { spark.conf.set(...max-listing-depth..., String.valueOf(depth + 1)); }

When it happens

Trigger: Running DeleteOrphanFiles on a table whose directory tree under the location is deeper than MAX_EXECUTOR_LISTING_DEPTH, so the Listing iterator still has subDirs after traversing the allowed depth.

Common situations: Very deep partitioning schemes (e.g. year/month/day/hour/... plus app-level folders); pointing the action at a location containing unrelated nested directory trees.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Cannot clean files incrementally when snapshot IDs are speci
• Invalid codec name: %s
• Class %s does not implement DynamicRecordGeneratorSQL
• Failed to instantiate DynamicRecordGeneratorSQL %s
• Cannot parse order: parser is not an Iceberg ExtendedParser

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d5be54e484805e8b. Report an issue: GitHub.