{"record":{"id":"b229e9be5f33e726","repo":"apache/hadoop","slug":"failed-to-get-files-with-active-leases","errorCode":null,"errorMessage":"Failed to get files with active leases","messagePattern":"Failed to get files with active leases","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java","lineNumber":257,"sourceCode":"                !inodesInPath.isDescendant(ancestorDir)) {\n              continue;\n            }\n            iNodesInPaths.add(inodesInPath);\n          }\n          return iNodesInPaths;\n        }\n      };\n\n      // Submit the inode filter task to the Executor Service\n      futureList.add(inodeFilterService.submit(c));\n    }\n    inodeFilterService.shutdown();\n\n    for (Future<List<INodesInPath>> f : futureList) {\n      try {\n        iipSet.addAll(f.get());\n      } catch (Exception e) {\n        throw new IOException(\"Failed to get files with active leases\", e);\n      }\n    }\n    final long endTimeMs = Time.monotonicNow();\n    if ((endTimeMs - startTimeMs) > 1000) {\n      LOG.info(\"Took {} ms to collect {} open files with leases {}\",\n          (endTimeMs - startTimeMs), iipSet.size(), ((ancestorDir != null) ?\n              \" under \" + ancestorDir.getFullPathName() : \".\"));\n    }\n    return iipSet;\n  }\n\n  public BatchedListEntries<OpenFileEntry> getUnderConstructionFiles(\n      final long prevId) throws IOException {\n    return getUnderConstructionFiles(prevId,\n        OpenFilesIterator.FILTER_PATH_DEFAULT);\n  }\n\n  /**","sourceCodeStart":239,"sourceCodeEnd":275,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java#L239-L275","documentation":"LeaseManager.listOpenFiles fans out inode-filter tasks over an ExecutorService and merges the Futures into one result set; if any f.get() throws (ExecutionException from inside a worker, or interruption) while collecting open-for-write paths with active leases, the whole listing aborts with this wrapper. The attached cause carries the real failure from the worker thread, not from the lease scan itself.","triggerScenarios":"Invoking the open-files listing (hdfs dfsadmin -openFiles, the OpenFileEntry batch fetch, or getUnderConstructionFiles paths) while a worker throws - namespace/inode-map inconsistency, an executor shutdown race when the NN is stopping (tasks submitted then service shut down before get), or interrupted caller thread.","commonSituations":"Listing issued during NN shutdown or safemode transitions; very large open-file counts under memory pressure; GC pauses making tasks fail; custom inode filter extensions throwing.","solutions":["Inspect the cause: RejectedExecutionException/InterruptedException points to a shutdown race, an NPE in the filter points to a namespace inconsistency.","Retry the listing once the NN is stable (safemode exited, no restart in progress) - transient races are common.","If it fails deterministically for one path, narrow with the ancestorDir filter and inspect that subtree's leases via hdfs fsck / lease recovery.","Check NN memory and GC logs - long pauses increase task failure rates during big scans."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Cheap pre-check: skip the listing while the namespace is still loading\nif (namesystem.isInSafeMode() || namesystem.isInStandbyState() && !isPopulated()) {\n  LOG.info(\"Deferring open-files listing until namespace is stable\");\n  return Collections.emptySet();\n}","typeGuard":null,"tryCatchPattern":"for (int attempt = 1; attempt <= 3; attempt++) {\n  try {\n    return leaseManager.listOpenFiles(ancestorDir, ecPaths);\n  } catch (IOException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"Failed to get files with active leases\")\n        && attempt < 3) {\n      Thread.sleep(1000L * attempt); // worker raced NN state change; back off and retry\n      continue;\n    }\n    throw e;\n  }\n}","preventionTips":["Run open-files listings when the NN is steady: safemode exited, no restart or big upgrade in flight.","Chunk huge listings with the batched API instead of one giant scan.","Keep NN GC and memory healthy - long pauses raise task failure rates during fan-out scans."],"tags":["hdfs","leases","concurrency","executor","open-files"],"backgroundTag":"executor-task-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}