apache/hadoop · error · FatalRpcServerException

FATAL_DESERIALIZING_REQUEST

FATAL_DESERIALIZING_REQUEST

Error message

"IPC server unable to read call parameters: " + t.getMessage()

What it means

The server recognized the request wrapper class but buffer.newInstance() threw while deserializing the call parameters (the protobuf request message). The original failure is logged with the client address, protocol, and rpcKind, then rethrown as FATAL_DESERIALIZING_REQUEST. It almost always means client and server disagree on the protobuf schema of a method's parameters.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java:2978

      }
      Class<? extends Writable> rpcRequestClass =
          getRpcRequestWrapper(header.getRpcKind());
      if (rpcRequestClass == null) {
        LOG.warn("Unknown rpc kind {} from client {}", header.getRpcKind(), getHostAddress());
        throw new FatalRpcServerException(
            RpcErrorCodeProto.FATAL_INVALID_RPC_HEADER,
            "Unknown rpc kind in rpc header " + header.getRpcKind());
      }
      Writable rpcRequest;
      try { //Read the rpc request
        rpcRequest = buffer.newInstance(rpcRequestClass, conf);
      } catch (RpcServerException rse) { // lets tests inject failures.
        throw rse;
      } catch (Throwable t) { // includes runtime exception from newInstance
        LOG.warn(
            "Unable to read call parameters for client {} on connection protocol {} for rpcKind {}",
            getHostAddress(), this.protocolName, header.getRpcKind(), t);
        throw new FatalRpcServerException(
            RpcErrorCodeProto.FATAL_DESERIALIZING_REQUEST,
            "IPC server unable to read call parameters: " + t.getMessage());
      }

      Span span = null;
      if (header.hasTraceInfo()) {
        RPCTraceInfoProto traceInfoProto = header.getTraceInfo();
        if (traceInfoProto.hasSpanContext()) {
          if (tracer == null) {
            setTracer(Tracer.curThreadTracer());
          }
          if (tracer != null) {
            // If the incoming RPC included tracing info, always continue the
            // trace
            SpanContext spanCtx = TraceUtils.byteStringToSpanContext(
                traceInfoProto.getSpanContext());
            if (spanCtx != null) {
              span = tracer.newSpan(

View on GitHub (pinned to 2add963021)

Solutions

1. Align hadoop-client artifact versions on the client with the server and eliminate duplicate hadoop jars from the classpath
2. For custom protocols, regenerate protobuf classes from the same .proto revision on both sides and keep @ProtocolInfo versionIDs in sync
3. Read the underlying cause in the server WARN log ('Unable to read call parameters') to identify which message failed to decode
4. During rolling upgrades, finish the upgrade so no node pairs speak different protocol revisions

// pre-flight: verify protocol versions before first call
long serverVersion = proxy.getProtocolVersion(MyProtocol.class.getName(),
    MyProtocol.versionID);
if (serverVersion != MyProtocol.versionID) {
  throw new IllegalStateException("Protocol version skew: client "
      + MyProtocol.versionID + " vs server " + serverVersion);
}

try {
  proxy.call(args);
} catch (RemoteException re) {
  if (re.getMessage().contains("unable to read call parameters")) {
    // schema mismatch: align hadoop/protocol jars; retrying will not fix it
    LOG.error("Request deserialization failed; check jar versions", re);
    throw new IllegalStateException("Client/server protobuf schema mismatch", re);
  } else { throw re; }
}

When it happens

Trigger: A method signature or protobuf message changed on one side (field added/removed/renumbered incompatibly); duplicate or mismatched hadoop-client jars on either end; a protocol implementation not matching the interface version the other side compiled against.

Common situations: Rolling upgrades where an old client calls a new server with changed method params; applications shading hadoop with relocators while the server expects unshaded protos; custom protocols whose .proto files drifted between producer and consumer; truncated payloads from network issues.

Related errors

• Unknown method {} called on {} protocol.
• Unknown method ${methodName} called on ${connectionProtocolN
• Serverside implements {}. The following requested protocol i
• Serverside implements {}. The following requested protocol i
• Too many or few parameters for request. Method: [{}], Expect

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1bb1b2689dc1c0c1. Report an issue: GitHub.