apache/hadoop · error · RpcClientException

RPC response length mismatch

Error message

RPC response length mismatch

What it means

After deserializing a SUCCESS response, the client verifies the RPC packet is fully consumed: if packet.remaining() > 0, the server sent more bytes than the client's Writable value readFields() consumed, so the two sides disagree on the response encoding. It throws RpcClientException('RPC response length mismatch') because continuing would desynchronize the connection.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java:1271

            packet.getValue(RpcResponseHeaderProto.getDefaultInstance());
        checkResponse(header);

        int callId = header.getCallId();
        if (LOG.isDebugEnabled())
          LOG.debug(getName() + " got value #" + callId);

        RpcStatusProto status = header.getStatus();
        if (status == RpcStatusProto.SUCCESS) {
          Writable value = packet.newInstance(valueClass, conf);
          final Call call = calls.remove(callId);
          if (call.alignmentContext != null) {
            call.alignmentContext.receiveResponseState(header);
          }
          call.setRpcResponse(value);
        }
        // verify that packet length was correct
        if (packet.remaining() > 0) {
          throw new RpcClientException("RPC response length mismatch");
        }
        if (status != RpcStatusProto.SUCCESS) { // Rpc Request failed
          final String exceptionClassName = header.hasExceptionClassName() ?
                header.getExceptionClassName() : 
                  "ServerDidNotSetExceptionClassName";
          final String errorMsg = header.hasErrorMsg() ? 
                header.getErrorMsg() : "ServerDidNotSetErrorMsg" ;
          final RpcErrorCodeProto erCode = 
                    (header.hasErrorDetail() ? header.getErrorDetail() : null);
          if (erCode == null) {
             LOG.warn("Detailed error code not set by server on rpc error");
          }
          RemoteException re = new RemoteException(exceptionClassName, errorMsg, erCode);
          if (status == RpcStatusProto.ERROR) {
            final Call call = calls.remove(callId);
            call.setException(re);
          } else if (status == RpcStatusProto.FATAL) {
            // Close the connection

View on GitHub (pinned to 2add963021)

Solutions

1. Pin client and server to compatible Hadoop versions for the RPC interface in question.
2. For custom Writable payloads, audit readFields() to consume exactly the bytes write() emits (symmetry test: write then read, assert buffer drained).
3. Capture the mismatched call (turn on org.apache.hadoop.ipc DEBUG) to identify which method's response overflows.
4. If mid-rolling-upgrade, complete it or use the version-specific proxy factories Hadoop provides.

Example fix

// before: custom Writable under-reads
@Override
public void readFields(DataInput in) throws IOException {
  this.name = in.readUTF(); // write() also wrote a timestamp that is never read -> length mismatch
}

// after
@Override
public void readFields(DataInput in) throws IOException {
  this.name = in.readUTF();
  this.timestamp = in.readLong(); // mirror write() exactly
}

// for custom Writable responses, assert symmetric serialization before deploying:
MyValue v = sampleValue();
DataOutputBuffer out = new DataOutputBuffer();
v.write(out);
DataInputBuffer in = new DataInputBuffer();
in.reset(out.getData(), 0, out.getLength());
MyValue copy = new MyValue();
copy.readFields(in);
assert in.available() == 0 : "readFields() must consume exactly what write() produced";

try {
  call();
} catch (RpcClientException e) {
  if ("RPC response length mismatch".equals(e.getMessage())) {
    // serialization disagreement: pin client/server versions; do NOT retry on this connection
    throw new IllegalStateException("client/server RPC serialization mismatch", e);
  }
  throw e;
}

When it happens

Trigger: Client and server running Hadoop versions whose protobuf/wire formats differ for that call; a custom Writable response whose readFields() under-reads what write() produced (fields skipped); enum/optional-field drift after a partial upgrade; intermediary corrupting framing.

Common situations: Mixed-version clusters during rolling upgrades; custom RPC endpoints with hand-written Writable serialization that is not symmetric; shading/relocation creating two incompatible serializer versions; rare JDK serialization differences.

Related errors

• Client IDs not matched: local ID={}, ID in response={}
• Unknown protocol: {}
• Serverside implements {}. The following requested protocol i
• Serverside implements {}. The following requested protocol i
• Class {} already registered but maps to {} and not {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d79d875ac6a6403c. Report an issue: GitHub.