apache/iceberg · error · RuntimeException

Could not list sub directories, reached maximum depth:

Error message

Could not list sub directories, reached maximum depth:

What it means

The distributed listing task in DeleteOrphanFilesSparkAction recurses into subdirectories up to MAX_EXECUTOR_LISTING_DEPTH (2000). If after the maximum depth there are still unexplored subdirectories, it throws this RuntimeException instead of silently returning an incomplete orphan-file list, because a truncated listing could cause live files to be judged orphan and deleted.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:518

      List<String> subDirs = Lists.newArrayList();
      List<String> files = Lists.newArrayList();

      Predicate<FileStatus> predicate = file -> file.getModificationTime() < olderThanTimestamp;

      while (dirs.hasNext()) {
        FileSystemWalker.listDirRecursivelyWithHadoop(
            dirs.next(),
            specs,
            predicate,
            hadoopConf.value().value(),
            MAX_EXECUTOR_LISTING_DEPTH,
            MAX_EXECUTOR_LISTING_DIRECT_SUB_DIRS,
            subDirs::add,
            files::add);
      }

      if (!subDirs.isEmpty()) {
        throw new RuntimeException(
            "Could not list sub directories, reached maximum depth: " + MAX_EXECUTOR_LISTING_DEPTH);
      }

      return files.iterator();
    }
  }

  private static class FindOrphanFiles
      implements MapPartitionsFunction<Tuple2<FileURI, FileURI>, String> {

    private final PrefixMismatchMode mode;
    private final SetAccumulator<Pair<String, String>> conflicts;

    FindOrphanFiles(PrefixMismatchMode mode, SetAccumulator<Pair<String, String>> conflicts) {
      this.mode = mode;
      this.conflicts = conflicts;
    }

View on GitHub (pinned to 86d9c8fc54)

Solutions

Flatten the physical directory structure so nesting is far below the 2000-level cap.
If the tree is legitimately deep, list in stages: run the action per sub-prefix (olderLocation/location) and merge results manually.
Investigate the directory tree for accidental runaway nesting (e.g. a job writing timestamps-as-nested-dirs) and clean it up with non-destructive tooling first.
Increase MAX_EXECUTOR_LISTING_DEPTH in a patched build only if you fully understand the orphan-detection risk of deeper listings.

Defensive patterns

Strategy: validation

Validate before calling

// Before running, sanity-check nesting depth of the table location
long maxDepth = java.nio.file.Files.walk(dir).filter(Files::isDirectory)
    .mapToLong(p -> p.getNameCount()).max().orElse(0);
if (maxDepth > 2000) throw new IllegalStateException("Listing depth " + maxDepth + " exceeds executor cap");

Try / catch

try {
  action.execute();
} catch (RuntimeException e) {
  if (e.getMessage().startsWith("Could not list sub directories, reached maximum depth")) {
    throw new IllegalStateException("Table location too deeply nested; flatten before orphan cleanup", e);
  } else throw e;
}

Prevention

Avoid generating data locations with timestamp-per-directory hierarchies that nest unboundedly.
Periodically audit table location depth with a dry listing before orphan-file deletion.
Run orphan-file cleanup scoped to shallow prefixes rather than one giant deep root.

When it happens

Trigger: Running deleteOrphanFiles on a location with directory nesting deeper than 2000 levels (or a listing that expands beyond the depth cap), e.g. deeply hierarchical data layouts or pathological directory structures in the table location.

Common situations: Object-store buckets with extremely deep prefixes (each '/' counted), corrupted/malicious directory trees, or migrations from systems that created very deep folder hierarchies.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/2533fd80e3d4c3ad. Report an issue: GitHub.