apache/hadoop · error · RpcException

RPC response has a length of %d exceeds maximum data length

Error message

RPC response has a length of %d exceeds maximum data length

What it means

Client-side guard in IpcStreams.readResponse: every connection caches ipc.maximum.response.length (CommonConfigurationKeys.IPC_MAXIMUM_RESPONSE_LENGTH, default 2146435072 bytes); when a response frame's declared length exceeds that cap the client refuses to allocate and throws RpcException instead of attempting a huge allocation. It protects the client JVM from OOME when a single RPC response (e.g., a huge directory listing) is larger than allowed.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java:1963

    }

    public ByteBuffer readResponse() throws IOException {
      int length = in.readInt();
      if (firstResponse) {
        firstResponse = false;
        // pre-rpcv9 exception, almost certainly a version mismatch.
        if (length == -1) {
          in.readInt(); // ignore fatal/error status, it's fatal for us.
          throw new RemoteException(WritableUtils.readString(in),
                                    WritableUtils.readString(in));
        }
      }
      if (length <= 0) {
        throw new RpcException(String.format("RPC response has " +
            "invalid length of %d", length));
      }
      if (maxResponseLength > 0 && length > maxResponseLength) {
        throw new RpcException(String.format("RPC response has a " +
            "length of %d exceeds maximum data length", length));
      }
      ByteBuffer bb = ByteBuffer.allocate(length);
      in.readFully(bb.array());
      return bb;
    }

    public void sendRequest(byte[] buf) throws IOException {
      out.write(buf);
    }

    @Override
    public void flush() throws IOException {
      out.flush();
    }

    @Override
    public void close() {

View on GitHub (pinned to 2add963021)

Solutions

Raise the cap on the CLIENT configuration: set ipc.maximum.response.length to a larger value (bytes) in the client's Configuration before creating the proxy/FileSystem.
Better: reduce per-call response size — paginate directory listings (iterate with listStatusIterator / listLocatedStatus, or use HDFS listing batch APIs) instead of one giant call.
Confirm which call produces the fat response from the server logs/metrics, and split the workload.

Example fix

// before
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(uri, conf);
FileStatus[] all = fs.listStatus(hugeDir); // single multi-GB response -> RpcException

// after
Configuration conf = new Configuration();
conf.setInt("ipc.maximum.response.length", 512 * 1024 * 1024); // 512MB cap
RemoteIterator<FileStatus> it = fs.listStatusIterator(hugeDir); // server-side paging
while (it.hasNext()) { process(it.next()); }

Defensive patterns

Strategy: validation

Validate before calling

// client-side, before creating the proxy/FileSystem
int expectedMaxBytes = 512 * 1024 * 1024;
if (conf.getInt("ipc.maximum.response.length", Integer.MAX_VALUE) < expectedMaxBytes) {
  conf.setInt("ipc.maximum.response.length", expectedMaxBytes);
}
// and prefer chunked iteration for big listings
if (fs.listStatusIterator != null) { /* use iterator APIs */ }

Try / catch

try {
  return proxy.bulkList(req);
} catch (RpcException e) {
  if (e.getMessage() != null && e.getMessage().contains("exceeds maximum data length")) {
    throw new IllegalArgumentException("Response too large; split the request or raise ipc.maximum.response.length", e);
  }
  throw e;
}

Prevention

Use paging/iterator APIs (listStatusIterator, listing batches) for unbounded result sets.
Set ipc.maximum.response.length deliberately on clients that legitimately fetch large payloads.
Log response sizes in custom RPC servers to catch payload growth before clients do.

When it happens

Trigger: listStatus/getBlockLocations/listLocatedStatus on a directory with millions of entries; a custom RPC returning a very large protobuf/Writable payload; server returning a large error snapshot (e.g., StandbyException payloads are small, but fat responses like fsimage transfers over RPC are not typical) — any single response frame over the configured cap.

Common situations: Clients enumerating massive directories (spark/hive partition scans, distcp building file lists); default cap lowered deliberately to protect clients; after server upgrades that batch more data per call.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0d238775f6d945b1. Report an issue: GitHub.