{"record":{"id":"880c15eb9506269f","repo":"apache/hadoop","slug":"rpc-response-has-invalid-length-of-d","errorCode":null,"errorMessage":"RPC response has invalid length of %d","messagePattern":"RPC response has invalid length of (.+?)","errorType":"exception","errorClass":"RpcException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java","lineNumber":1959,"sourceCode":"\n    private void setOutputStream(OutputStream os) {\n      this.out = (os instanceof DataOutputStream)\n          ? (DataOutputStream)os : new DataOutputStream(os);\n    }\n\n    public ByteBuffer readResponse() throws IOException {\n      int length = in.readInt();\n      if (firstResponse) {\n        firstResponse = false;\n        // pre-rpcv9 exception, almost certainly a version mismatch.\n        if (length == -1) {\n          in.readInt(); // ignore fatal/error status, it's fatal for us.\n          throw new RemoteException(WritableUtils.readString(in),\n                                    WritableUtils.readString(in));\n        }\n      }\n      if (length <= 0) {\n        throw new RpcException(String.format(\"RPC response has \" +\n            \"invalid length of %d\", length));\n      }\n      if (maxResponseLength > 0 && length > maxResponseLength) {\n        throw new RpcException(String.format(\"RPC response has a \" +\n            \"length of %d exceeds maximum data length\", length));\n      }\n      ByteBuffer bb = ByteBuffer.allocate(length);\n      in.readFully(bb.array());\n      return bb;\n    }\n\n    public void sendRequest(byte[] buf) throws IOException {\n      out.write(buf);\n    }\n\n    @Override\n    public void flush() throws IOException {\n      out.flush();","sourceCodeStart":1941,"sourceCodeEnd":1977,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1941-L1977","documentation":"IpcStreams.readResponse reads a 4-byte framed RPC response; after the first-response -1 handshake-error case is handled, any length <= 0 throws RpcException('RPC response has invalid length of N'). A non-positive frame length is impossible in the protocol, so the byte stream is no longer aligned with the framing — most commonly a client/server Hadoop version or wire-protocol mismatch, or a corrupted/garbage stream (e.g., an HTTP proxy or wrong service answering on the port).","triggerScenarios":"A Hadoop client whose IPC wire version differs from the server's answers with a non-framed payload; connecting an RPC client to a port served by a different protocol (HTTP server, HTTPS endpoint); a truncated/corrupted TCP stream after a mid-read network fault being read as the next frame.","commonSituations":"Mixing Hadoop versions across a cluster upgrade (e.g., 2.x client vs 3.x server on a non-compatible protocol); connecting to a webhdfs/HTTP port with an RPC client; JVM/serializer divergence on custom Writables; misconfigured port in fs.defaultFS or yarn.resourcemanager.address.","solutions":["Verify the endpoint: confirm the address/port is a real Hadoop RPC server (e.g., dfs.namenode.rpc-address, not the http one).","Align client and server Hadoop versions/protocol implementations (same artifacts for both sides of any custom protocol).","Close the proxy and reconnect with a fresh connection so framing resynchronizes; if it recurs, capture the first bytes received to identify what is actually answering."],"exampleFix":"// before\nConfiguration conf = new Configuration();\nconf.set(\"fs.defaultFS\", \"hdfs://nn:9870\"); // 9870 is the HTTP port\nFileSystem fs = FileSystem.get(conf); // RPC framing broken -> invalid length\n\n// after\nConfiguration conf = new Configuration();\nconf.set(\"fs.defaultFS\", \"hdfs://nn:8020\"); // the RPC port (dfs.namenode.rpc-address)\nFileSystem fs = FileSystem.get(conf);","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  return proxy.call(req);\n} catch (RpcException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"invalid length\")) {\n    // framing desync: discard the proxy/connection entirely\n    RPC.stopProxy(proxy);\n    proxy = buildFreshProxy();\n    throw new IllegalStateException(\"RPC wire mismatch - verify endpoint and versions\", e);\n  }\n  throw e;\n}","preventionTips":["Pin client and server Hadoop versions during upgrades; test cross-version RPC in staging.","Smoke-test endpoints with a trivial RPC (e.g., ServerDefaults) before heavy traffic to surface protocol mismatch early.","Never point RPC clients at HTTP/HTTPS ports; keep address configs sourced from the *-rpc-address keys."],"tags":["hadoop","ipc","rpc","protocol-mismatch","wire-format","corruption"],"backgroundTag":"rpc-framing-corruption","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}