apache/hadoop · error · IOException

"Unexpected data length " + dataLength + "!! from " + getHos

Error message

"Unexpected data length " + dataLength + "!! from " + getHostAddress()

What it means

Every Hadoop IPC frame starts with a 4-byte big-endian length. checkDataLength() rejects any frame whose declared length is negative, which cannot happen in a valid RPC stream, so it almost always means the bytes on the socket are not Hadoop IPC at all (wrong protocol, TLS to a plaintext port, or stream corruption). The offending connection is closed after a WARN log.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java:2518

    }

    private void disposeSasl() {
      if (saslServer != null) {
        try {
          saslServer.dispose();
        } catch (SaslException ignored) {
        } finally {
          saslServer = null;
        }
      }
    }

    private void checkDataLength(int dataLength) throws IOException {
      if (dataLength < 0) {
        String error = "Unexpected data length " + dataLength +
                       "!! from " + getHostAddress();
        LOG.warn(error);
        throw new IOException(error);
      } else if (dataLength > maxDataLength) {
        String error = "Requested data length " + dataLength +
              " is longer than maximum configured RPC length " + 
            maxDataLength + ".  RPC came from " + getHostAddress();
        LOG.warn(error);
        throw new IOException(error);
      }
    }

    /**
     * This method reads in a non-blocking fashion from the channel: 
     * this method is called repeatedly when data is present in the channel; 
     * when it has enough data to process one rpc it processes that rpc.
     * 
     * On the first pass, it processes the connectionHeader, 
     * connectionContext (an outOfBand RPC) and at most one RPC request that 
     * follows that. On future passes it will process at most one RPC request.
     *

View on GitHub (pinned to 2add963021)

Solutions

Verify the client connects to the true RPC port (e.g. NameNode rpc-address, not the http-address) and that the host address comes from a verified source
Ensure TLS/SSL is either enabled on both client and server or on neither for that port
Check what is actually connecting: the log line prints getHostAddress() of the offending peer
Remove any load balancer, sidecar, or proxy that forwards non-IPC traffic to the RPC listener

Defensive patterns

Strategy: try-catch

Validate before calling

// before opening a connection, sanity-check the endpoint is an IPC port
InetSocketAddress rpcAddr = NetUtils.createSocketAddr(
    conf.get("fs.defaultFS"), 8020, "fs.defaultFS");

Try / catch

try {
  proxy = RPC.getProxy(protocol, versionID, addr, conf);
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("Unexpected data length")) {
    // peer is not speaking Hadoop IPC on this port (e.g. TLS to plaintext port)
    LOG.error("{} is not a Hadoop RPC endpoint; check scheme/port", addr, e);
  }
  throw e;
}

Prevention

Derive RPC addresses from verified config keys, never hardcode
Keep TLS listeners and plaintext RPC listeners on distinct, documented ports
Exclude RPC ports from load-balancer and health-check pools

When it happens

Trigger: An HTTPS/TLS client or curl https:// connects to the unencrypted RPC port and the TLS bytes decode to a negative integer; a garbage payload from a port scanner; a client speaking HTTP, Redis, or any other protocol to the RPC listener; memory/socket corruption mid-stream.

Common situations: Misconfigured service discovery or load balancer routing TLS traffic to the IPC port; operators testing connectivity with netcat/HTTP clients against the RPC port; middleboxes or proxies rewriting traffic; mixing up RPC and HTTP ports in configuration (e.g. pointing fs.defaultFS at the wrong port).

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/80c270e30299fd41. Report an issue: GitHub.