apache/hadoop · error · IllegalArgumentException

Too many source paths (%d > %d)

Error message

Too many source paths (%d > %d)

What it means

getBatchedListing rejects requests whose srcs array is larger than the configured batchedListingLimit (dfs.namenode.batchedListing.limit, default 100). IllegalArgumentException reports both counts, guarding the NameNode from unbounded listing work per request.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:4310

  public byte[] getSrcPathsHash(String[] srcs) {
    synchronized (digest) {
      for (String src : srcs) {
        digest.update(src.getBytes(StandardCharsets.UTF_8));
      }
      byte[] result = digest.digest();
      digest.reset();
      return result;
    }
  }

  BatchedDirectoryListing getBatchedListing(String[] srcs, byte[] startAfter,
      boolean needLocation) throws IOException {

    if (srcs.length > this.batchedListingLimit) {
      String msg = String.format("Too many source paths (%d > %d)",
          srcs.length, batchedListingLimit);
      throw new IllegalArgumentException(msg);
    }

    // Parse the startAfter key if present
    int srcsIndex = 0;
    byte[] indexStartAfter = new byte[0];

    if (startAfter.length > 0) {
      BatchedListingKeyProto startAfterProto =
          BatchedListingKeyProto.parseFrom(startAfter);
      // Validate that the passed paths match the checksum from key
      Preconditions.checkArgument(
          Arrays.equals(
              startAfterProto.getChecksum().toByteArray(),
              getSrcPathsHash(srcs)));
      srcsIndex = startAfterProto.getPathIndex();
      indexStartAfter = startAfterProto.getStartAfter().toByteArray();
      // Special case: if the indexStartAfter key is an empty array, it
      // means the last element we listed was a file, not a directory.

View on GitHub (pinned to 2add963021)

Solutions

1. Chunk the request client-side so each call stays below the limit (default 100 paths)
2. Or raise dfs.namenode.batchedListing.limit in hdfs-site.xml on the NameNode and restart it
3. Query the effective limit from config before building large requests

Example fix

// before
BatchedDirectoryListing r = nn.getBatchedListing(allSrcs, startAfter, needLocation);

// after
int LIMIT = conf.getInt(DFSConfigKeys.DFS_NAMENODE_BATCHED_LISTING_LIMIT_KEY,
                         DFSConfigKeys.DFS_NAMENODE_BATCHED_LISTING_LIMIT_DEFAULT);
for (List<String> chunk : Lists.partition(allSrcs, LIMIT)) {
  BatchedDirectoryListing r = nn.getBatchedListing(chunk.toArray(new String[0]), startAfter, needLocation);
}

int limit = conf.getInt(
    DFSConfigKeys.DFS_NAMENODE_BATCHED_LISTING_LIMIT_KEY,
    DFSConfigKeys.DFS_NAMENODE_BATCHED_LISTING_LIMIT_DEFAULT); // 100
for (List<String> chunk : Lists.partition(srcs, limit)) {
  listBatched(chunk.toArray(new String[0]));
}

When it happens

Trigger: A single batched listing call (listStatusBatched / BatchedDirectoryListing RPC) submitting more source paths than the limit, e.g. enumerating hundreds of directories in one request.

Common situations: Clients batching many directory listings per call for latency reasons; the limit lowered by an administrator; code written against a smaller test cluster moved to production lists.

Related errors

• Path {} does not exist
• Unknown nameservice: {}
• Configuration has multiple addresses that match local node's
• Cannot delete/rename non-empty protected directory {}
• Cannot delete/rename non-empty protected subdirectory {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/68a38e74f3210035. Report an issue: GitHub.