What causes "Failed to get files with active leases" in apache/hadoop?

Inspect the cause: RejectedExecutionException/InterruptedException points to a shutdown race, an NPE in the filter points to a namespace inconsistency.

apache/hadoop · error · IOException

Failed to get files with active leases

Error message

Failed to get files with active leases

What it means

LeaseManager.listOpenFiles fans out inode-filter tasks over an ExecutorService and merges the Futures into one result set; if any f.get() throws (ExecutionException from inside a worker, or interruption) while collecting open-for-write paths with active leases, the whole listing aborts with this wrapper. The attached cause carries the real failure from the worker thread, not from the lease scan itself.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java:257

                !inodesInPath.isDescendant(ancestorDir)) {
              continue;
            }
            iNodesInPaths.add(inodesInPath);
          }
          return iNodesInPaths;
        }
      };

      // Submit the inode filter task to the Executor Service
      futureList.add(inodeFilterService.submit(c));
    }
    inodeFilterService.shutdown();

    for (Future<List<INodesInPath>> f : futureList) {
      try {
        iipSet.addAll(f.get());
      } catch (Exception e) {
        throw new IOException("Failed to get files with active leases", e);
      }
    }
    final long endTimeMs = Time.monotonicNow();
    if ((endTimeMs - startTimeMs) > 1000) {
      LOG.info("Took {} ms to collect {} open files with leases {}",
          (endTimeMs - startTimeMs), iipSet.size(), ((ancestorDir != null) ?
              " under " + ancestorDir.getFullPathName() : "."));
    }
    return iipSet;
  }

  public BatchedListEntries<OpenFileEntry> getUnderConstructionFiles(
      final long prevId) throws IOException {
    return getUnderConstructionFiles(prevId,
        OpenFilesIterator.FILTER_PATH_DEFAULT);
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

Inspect the cause: RejectedExecutionException/InterruptedException points to a shutdown race, an NPE in the filter points to a namespace inconsistency.
Retry the listing once the NN is stable (safemode exited, no restart in progress) - transient races are common.
If it fails deterministically for one path, narrow with the ancestorDir filter and inspect that subtree's leases via hdfs fsck / lease recovery.
Check NN memory and GC logs - long pauses increase task failure rates during big scans.

Defensive patterns

Strategy: retry

Validate before calling

// Cheap pre-check: skip the listing while the namespace is still loading
if (namesystem.isInSafeMode() || namesystem.isInStandbyState() && !isPopulated()) {
  LOG.info("Deferring open-files listing until namespace is stable");
  return Collections.emptySet();
}

Try / catch

for (int attempt = 1; attempt <= 3; attempt++) {
  try {
    return leaseManager.listOpenFiles(ancestorDir, ecPaths);
  } catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("Failed to get files with active leases")
        && attempt < 3) {
      Thread.sleep(1000L * attempt); // worker raced NN state change; back off and retry
      continue;
    }
    throw e;
  }
}

Prevention

Run open-files listings when the NN is steady: safemode exited, no restart or big upgrade in flight.
Chunk huge listings with the batched API instead of one giant scan.
Keep NN GC and memory healthy - long pauses raise task failure rates during fan-out scans.

When it happens

Trigger: Invoking the open-files listing (hdfs dfsadmin -openFiles, the OpenFileEntry batch fetch, or getUnderConstructionFiles paths) while a worker throws - namespace/inode-map inconsistency, an executor shutdown race when the NN is stopping (tasks submitted then service shut down before get), or interrupted caller thread.

Common situations: Listing issued during NN shutdown or safemode transitions; very large open-file counts under memory pressure; GC pauses making tasks fail; custom inode filter extensions throwing.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b229e9be5f33e726. Report an issue: GitHub.