apache/iceberg · error · java.lang.RuntimeException

Could not list sub directories, reached maximum depth:

Error message

Could not list sub directories, reached maximum depth: 

What it means

The executor-side listing task in DeleteOrphanFiles walks directories up to spark.sql.files.max... / MAX_EXECUTOR_LISTING_DEPTH levels. If subdirectories still remain at the maximum depth, it aborts with this RuntimeException rather than silently skipping files, preventing incomplete orphan detection.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:518

      List<String> subDirs = Lists.newArrayList();
      List<String> files = Lists.newArrayList();

      Predicate<FileStatus> predicate = file -> file.getModificationTime() < olderThanTimestamp;

      while (dirs.hasNext()) {
        FileSystemWalker.listDirRecursivelyWithHadoop(
            dirs.next(),
            specs,
            predicate,
            hadoopConf.value().value(),
            MAX_EXECUTOR_LISTING_DEPTH,
            MAX_EXECUTOR_LISTING_DIRECT_SUB_DIRS,
            subDirs::add,
            files::add);
      }

      if (!subDirs.isEmpty()) {
        throw new RuntimeException(
            "Could not list sub directories, reached maximum depth: " + MAX_EXECUTOR_LISTING_DEPTH);
      }

      return files.iterator();
    }
  }

  private static class FindOrphanFiles
      implements MapPartitionsFunction<Tuple2<FileURI, FileURI>, String> {

    private final PrefixMismatchMode mode;
    private final SetAccumulator<Pair<String, String>> conflicts;

    FindOrphanFiles(PrefixMismatchMode mode, SetAccumulator<Pair<String, String>> conflicts) {
      this.mode = mode;
      this.conflicts = conflicts;
    }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Increase the depth via the procedure's maxDepth / listing depth parameter (older_file... no; use max_depth option or the internal maxListingDepth override)
2. Point the action at a shallower location (e.g. a specific partition prefix) and run in batches
3. Restructure the data layout if deep nesting is accidental

Example fix

// before
CALL iceberg.system.remove_orphan_files(table => 'db.t') -- deeper than default depth
// after
CALL iceberg.system.remove_orphan_files(table => 'db.t', max_depth => 10);

// Estimate layout depth under the table location before listing
// hadoop fs -count on the location; ensure partition nesting < max listing depth

try { removeOrphanFiles(); } catch (RuntimeException e) { if (e.getMessage().startsWith("Could not list sub directories")) { /* increase depth or target a shallower prefix */ } }

When it happens

Trigger: Running remove_orphan_files on a table location whose directory tree is deeper than MAX_EXECUTOR_LISTING_DEPTH (default 3) beneath the listed root, and the parallel listing still finds subDirs at that depth.

Common situations: Very deep partitioning layouts (e.g. year/month/day/hour/...) or nested path conventions under the table location; partition columns beyond the depth limit.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Cannot remove orphan files with an interval less than 24 hou
• Could not list sub directories, reached maximum depth: ${MAX
• Cannot remove orphan files with an interval less than 24 hou
• Cannot remove orphan files with an interval less than 24 hou
• Unable to determine whether certain files are orphan. Metada

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/71c28a845bf67123. Report an issue: GitHub.