{"record":{"id":"68a38e74f3210035","repo":"apache/hadoop","slug":"too-many-source-paths-d-d","errorCode":null,"errorMessage":"Too many source paths (%d > %d)","messagePattern":"Too many source paths \\((.+?) > (.+?)\\)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java","lineNumber":4310,"sourceCode":"\n  public byte[] getSrcPathsHash(String[] srcs) {\n    synchronized (digest) {\n      for (String src : srcs) {\n        digest.update(src.getBytes(StandardCharsets.UTF_8));\n      }\n      byte[] result = digest.digest();\n      digest.reset();\n      return result;\n    }\n  }\n\n  BatchedDirectoryListing getBatchedListing(String[] srcs, byte[] startAfter,\n      boolean needLocation) throws IOException {\n\n    if (srcs.length > this.batchedListingLimit) {\n      String msg = String.format(\"Too many source paths (%d > %d)\",\n          srcs.length, batchedListingLimit);\n      throw new IllegalArgumentException(msg);\n    }\n\n    // Parse the startAfter key if present\n    int srcsIndex = 0;\n    byte[] indexStartAfter = new byte[0];\n\n    if (startAfter.length > 0) {\n      BatchedListingKeyProto startAfterProto =\n          BatchedListingKeyProto.parseFrom(startAfter);\n      // Validate that the passed paths match the checksum from key\n      Preconditions.checkArgument(\n          Arrays.equals(\n              startAfterProto.getChecksum().toByteArray(),\n              getSrcPathsHash(srcs)));\n      srcsIndex = startAfterProto.getPathIndex();\n      indexStartAfter = startAfterProto.getStartAfter().toByteArray();\n      // Special case: if the indexStartAfter key is an empty array, it\n      // means the last element we listed was a file, not a directory.","sourceCodeStart":4292,"sourceCodeEnd":4328,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L4292-L4328","documentation":"getBatchedListing rejects requests whose srcs array is larger than the configured batchedListingLimit (dfs.namenode.batchedListing.limit, default 100). IllegalArgumentException reports both counts, guarding the NameNode from unbounded listing work per request.","triggerScenarios":"A single batched listing call (listStatusBatched / BatchedDirectoryListing RPC) submitting more source paths than the limit, e.g. enumerating hundreds of directories in one request.","commonSituations":"Clients batching many directory listings per call for latency reasons; the limit lowered by an administrator; code written against a smaller test cluster moved to production lists.","solutions":["Chunk the request client-side so each call stays below the limit (default 100 paths)","Or raise dfs.namenode.batchedListing.limit in hdfs-site.xml on the NameNode and restart it","Query the effective limit from config before building large requests"],"exampleFix":"// before\nBatchedDirectoryListing r = nn.getBatchedListing(allSrcs, startAfter, needLocation);\n\n// after\nint LIMIT = conf.getInt(DFSConfigKeys.DFS_NAMENODE_BATCHED_LISTING_LIMIT_KEY,\n                         DFSConfigKeys.DFS_NAMENODE_BATCHED_LISTING_LIMIT_DEFAULT);\nfor (List<String> chunk : Lists.partition(allSrcs, LIMIT)) {\n  BatchedDirectoryListing r = nn.getBatchedListing(chunk.toArray(new String[0]), startAfter, needLocation);\n}","handlingStrategy":"validation","validationCode":"int limit = conf.getInt(\n    DFSConfigKeys.DFS_NAMENODE_BATCHED_LISTING_LIMIT_KEY,\n    DFSConfigKeys.DFS_NAMENODE_BATCHED_LISTING_LIMIT_DEFAULT); // 100\nfor (List<String> chunk : Lists.partition(srcs, limit)) {\n  listBatched(chunk.toArray(new String[0]));\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Read dfs.namenode.batchedListing.limit from config before building requests","Chunk large listings client-side regardless of the current limit","Prefer iterating one path per cursor-based listing for very large sets"],"tags":["hdfs","namenode","batched-listing","limit","input-validation"],"backgroundTag":"request-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}