{"record":{"id":"0d238775f6d945b1","repo":"apache/hadoop","slug":"rpc-response-has-a-length-of-d-exceeds-maximum-da","errorCode":null,"errorMessage":"RPC response has a length of %d exceeds maximum data length","messagePattern":"RPC response has a length of (.+?) exceeds maximum data length","errorType":"exception","errorClass":"RpcException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java","lineNumber":1963,"sourceCode":"    }\n\n    public ByteBuffer readResponse() throws IOException {\n      int length = in.readInt();\n      if (firstResponse) {\n        firstResponse = false;\n        // pre-rpcv9 exception, almost certainly a version mismatch.\n        if (length == -1) {\n          in.readInt(); // ignore fatal/error status, it's fatal for us.\n          throw new RemoteException(WritableUtils.readString(in),\n                                    WritableUtils.readString(in));\n        }\n      }\n      if (length <= 0) {\n        throw new RpcException(String.format(\"RPC response has \" +\n            \"invalid length of %d\", length));\n      }\n      if (maxResponseLength > 0 && length > maxResponseLength) {\n        throw new RpcException(String.format(\"RPC response has a \" +\n            \"length of %d exceeds maximum data length\", length));\n      }\n      ByteBuffer bb = ByteBuffer.allocate(length);\n      in.readFully(bb.array());\n      return bb;\n    }\n\n    public void sendRequest(byte[] buf) throws IOException {\n      out.write(buf);\n    }\n\n    @Override\n    public void flush() throws IOException {\n      out.flush();\n    }\n\n    @Override\n    public void close() {","sourceCodeStart":1945,"sourceCodeEnd":1981,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1945-L1981","documentation":"Client-side guard in IpcStreams.readResponse: every connection caches ipc.maximum.response.length (CommonConfigurationKeys.IPC_MAXIMUM_RESPONSE_LENGTH, default 2146435072 bytes); when a response frame's declared length exceeds that cap the client refuses to allocate and throws RpcException instead of attempting a huge allocation. It protects the client JVM from OOME when a single RPC response (e.g., a huge directory listing) is larger than allowed.","triggerScenarios":"listStatus/getBlockLocations/listLocatedStatus on a directory with millions of entries; a custom RPC returning a very large protobuf/Writable payload; server returning a large error snapshot (e.g., StandbyException payloads are small, but fat responses like fsimage transfers over RPC are not typical) — any single response frame over the configured cap.","commonSituations":"Clients enumerating massive directories (spark/hive partition scans, distcp building file lists); default cap lowered deliberately to protect clients; after server upgrades that batch more data per call.","solutions":["Raise the cap on the CLIENT configuration: set ipc.maximum.response.length to a larger value (bytes) in the client's Configuration before creating the proxy/FileSystem.","Better: reduce per-call response size — paginate directory listings (iterate with listStatusIterator / listLocatedStatus, or use HDFS listing batch APIs) instead of one giant call.","Confirm which call produces the fat response from the server logs/metrics, and split the workload."],"exampleFix":"// before\nConfiguration conf = new Configuration();\nFileSystem fs = FileSystem.get(uri, conf);\nFileStatus[] all = fs.listStatus(hugeDir); // single multi-GB response -> RpcException\n\n// after\nConfiguration conf = new Configuration();\nconf.setInt(\"ipc.maximum.response.length\", 512 * 1024 * 1024); // 512MB cap\nRemoteIterator<FileStatus> it = fs.listStatusIterator(hugeDir); // server-side paging\nwhile (it.hasNext()) { process(it.next()); }","handlingStrategy":"validation","validationCode":"// client-side, before creating the proxy/FileSystem\nint expectedMaxBytes = 512 * 1024 * 1024;\nif (conf.getInt(\"ipc.maximum.response.length\", Integer.MAX_VALUE) < expectedMaxBytes) {\n  conf.setInt(\"ipc.maximum.response.length\", expectedMaxBytes);\n}\n// and prefer chunked iteration for big listings\nif (fs.listStatusIterator != null) { /* use iterator APIs */ }","typeGuard":null,"tryCatchPattern":"try {\n  return proxy.bulkList(req);\n} catch (RpcException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"exceeds maximum data length\")) {\n    throw new IllegalArgumentException(\"Response too large; split the request or raise ipc.maximum.response.length\", e);\n  }\n  throw e;\n}","preventionTips":["Use paging/iterator APIs (listStatusIterator, listing batches) for unbounded result sets.","Set ipc.maximum.response.length deliberately on clients that legitimately fetch large payloads.","Log response sizes in custom RPC servers to catch payload growth before clients do."],"tags":["hadoop","ipc","rpc","response-size","limit","oom-protection"],"backgroundTag":"payload-too-large","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}