apache/hadoop · error · IOException

Client IDs not matched: local ID={}, ID in response={}

Error message

Client IDs not matched: local ID={}, ID in response={}

What it means

After checking the header is non-null, Client.checkResponse verifies that the clientId in the response header matches this client connection's own clientId (unless the server sent the DUMMY_CLIENT_ID). A mismatch means the response arriving on this connection belongs to a different client — the connection's framing/state is corrupted, or a proxy/server routed the response incorrectly. It throws IOException with both hex IDs for diagnosis.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java:315

  
  /**
   * Decrement this client's reference count
   */
  int decAndGetCount() {
    return refCount.decrementAndGet();
  }

  /** Check the rpc response header. */
  void checkResponse(RpcResponseHeaderProto header) throws IOException {
    if (header == null) {
      throw new EOFException("Response is null.");
    }
    if (header.hasClientId()) {
      // check client IDs
      final byte[] id = header.getClientId().toByteArray();
      if (!Arrays.equals(id, RpcConstants.DUMMY_CLIENT_ID)) {
        if (!Arrays.equals(id, clientId)) {
          throw new IOException("Client IDs not matched: local ID="
              + StringUtils.byteToHexString(clientId) + ", ID in response="
              + StringUtils.byteToHexString(header.getClientId().toByteArray()));
        }
      }
    }
  }

  Call createCall(RPC.RpcKind rpcKind, Writable rpcRequest) {
    return new Call(rpcKind, rpcRequest);
  }

  /** 
   * Class that represents an RPC call
   */
  static class Call {
    final int id;               // call id
    final int retry;           // retry count
    final Writable rpcRequest;  // the serialized rpc request

View on GitHub (pinned to 2add963021)

Solutions

1. Treat the connection as poisoned: close it (or let the Client discard it) and retry the call on a fresh connection.
2. Audit any custom code that shares or pools IPC connections across clients/threads incorrectly.
3. Compare the two hex IDs in the message — if the 'ID in response' repeats across hosts, a misbehaving middle layer is likely.
4. Verify client and server run compatible Hadoop versions after any partial upgrade.

try {
  call();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Client IDs not matched")) {
    // connection state is corrupted: force reconnect and retry; do not reuse the connection
    client.closeConnection();
    retryOnce();
  } else { throw e; }
}

When it happens

Trigger: RPC stream desynchronization after a partially handled error (client reads bytes offset from message boundaries); a buggy intermediary (proxy, router, test harness) writing another client's bytes onto this connection; bugs in custom RPC pipelines reusing connections across Client instances.

Common situations: Rare; appears when wrapping Hadoop RPC with custom connection pooling/multiplexing, in tests faking servers, or after version-mismatch desync events; usually accompanies other preceding IO errors on the same connection.

Related errors

• RPC response length mismatch
• Response is null.
• Exceeded limit of max asynchronous calls: %d, please configu
• connection has been closed
• Unknown protocol: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2859fe2a6725a7b6. Report an issue: GitHub.