{"record":{"id":"d79d875ac6a6403c","repo":"apache/hadoop","slug":"rpc-response-length-mismatch","errorCode":null,"errorMessage":"RPC response length mismatch","messagePattern":"RPC response length mismatch","errorType":"exception","errorClass":"RpcClientException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java","lineNumber":1271,"sourceCode":"            packet.getValue(RpcResponseHeaderProto.getDefaultInstance());\n        checkResponse(header);\n\n        int callId = header.getCallId();\n        if (LOG.isDebugEnabled())\n          LOG.debug(getName() + \" got value #\" + callId);\n\n        RpcStatusProto status = header.getStatus();\n        if (status == RpcStatusProto.SUCCESS) {\n          Writable value = packet.newInstance(valueClass, conf);\n          final Call call = calls.remove(callId);\n          if (call.alignmentContext != null) {\n            call.alignmentContext.receiveResponseState(header);\n          }\n          call.setRpcResponse(value);\n        }\n        // verify that packet length was correct\n        if (packet.remaining() > 0) {\n          throw new RpcClientException(\"RPC response length mismatch\");\n        }\n        if (status != RpcStatusProto.SUCCESS) { // Rpc Request failed\n          final String exceptionClassName = header.hasExceptionClassName() ?\n                header.getExceptionClassName() : \n                  \"ServerDidNotSetExceptionClassName\";\n          final String errorMsg = header.hasErrorMsg() ? \n                header.getErrorMsg() : \"ServerDidNotSetErrorMsg\" ;\n          final RpcErrorCodeProto erCode = \n                    (header.hasErrorDetail() ? header.getErrorDetail() : null);\n          if (erCode == null) {\n             LOG.warn(\"Detailed error code not set by server on rpc error\");\n          }\n          RemoteException re = new RemoteException(exceptionClassName, errorMsg, erCode);\n          if (status == RpcStatusProto.ERROR) {\n            final Call call = calls.remove(callId);\n            call.setException(re);\n          } else if (status == RpcStatusProto.FATAL) {\n            // Close the connection","sourceCodeStart":1253,"sourceCodeEnd":1289,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1253-L1289","documentation":"After deserializing a SUCCESS response, the client verifies the RPC packet is fully consumed: if packet.remaining() > 0, the server sent more bytes than the client's Writable value readFields() consumed, so the two sides disagree on the response encoding. It throws RpcClientException('RPC response length mismatch') because continuing would desynchronize the connection.","triggerScenarios":"Client and server running Hadoop versions whose protobuf/wire formats differ for that call; a custom Writable response whose readFields() under-reads what write() produced (fields skipped); enum/optional-field drift after a partial upgrade; intermediary corrupting framing.","commonSituations":"Mixed-version clusters during rolling upgrades; custom RPC endpoints with hand-written Writable serialization that is not symmetric; shading/relocation creating two incompatible serializer versions; rare JDK serialization differences.","solutions":["Pin client and server to compatible Hadoop versions for the RPC interface in question.","For custom Writable payloads, audit readFields() to consume exactly the bytes write() emits (symmetry test: write then read, assert buffer drained).","Capture the mismatched call (turn on org.apache.hadoop.ipc DEBUG) to identify which method's response overflows.","If mid-rolling-upgrade, complete it or use the version-specific proxy factories Hadoop provides."],"exampleFix":"// before: custom Writable under-reads\n@Override\npublic void readFields(DataInput in) throws IOException {\n  this.name = in.readUTF(); // write() also wrote a timestamp that is never read -> length mismatch\n}\n\n// after\n@Override\npublic void readFields(DataInput in) throws IOException {\n  this.name = in.readUTF();\n  this.timestamp = in.readLong(); // mirror write() exactly\n}","handlingStrategy":"validation","validationCode":"// for custom Writable responses, assert symmetric serialization before deploying:\nMyValue v = sampleValue();\nDataOutputBuffer out = new DataOutputBuffer();\nv.write(out);\nDataInputBuffer in = new DataInputBuffer();\nin.reset(out.getData(), 0, out.getLength());\nMyValue copy = new MyValue();\ncopy.readFields(in);\nassert in.available() == 0 : \"readFields() must consume exactly what write() produced\";","typeGuard":null,"tryCatchPattern":"try {\n  call();\n} catch (RpcClientException e) {\n  if (\"RPC response length mismatch\".equals(e.getMessage())) {\n    // serialization disagreement: pin client/server versions; do NOT retry on this connection\n    throw new IllegalStateException(\"client/server RPC serialization mismatch\", e);\n  }\n  throw e;\n}","preventionTips":["Keep client and server Hadoop versions compatible; complete rolling upgrades promptly.","Add write/read round-trip unit tests for every custom Writable.","Never leave unconsumed bytes in readFields().","Escalate immediately rather than retrying — this error poisons the connection's framing."],"tags":["hadoop","ipc","rpc-client","serialization","version-mismatch","protocol"],"backgroundTag":"protocol-version-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}