apache/hadoop · error · IOException

"Requested data length " + dataLength + " is longer than max

Error message

"Requested data length " + dataLength + " is longer than maximum configured RPC length " + maxDataLength + ".  RPC came from " + getHostAddress()

What it means

The server caps each RPC request payload at maxDataLength, configured via ipc.maximum.data.length (default 128 MB, see CommonConfigurationKeys). When a frame declares a length above that cap, checkDataLength() rejects it before allocating buffers, protecting the server from oversized or hostile requests. The connection is then closed.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java:2524

        } catch (SaslException ignored) {
        } finally {
          saslServer = null;
        }
      }
    }

    private void checkDataLength(int dataLength) throws IOException {
      if (dataLength < 0) {
        String error = "Unexpected data length " + dataLength +
                       "!! from " + getHostAddress();
        LOG.warn(error);
        throw new IOException(error);
      } else if (dataLength > maxDataLength) {
        String error = "Requested data length " + dataLength +
              " is longer than maximum configured RPC length " + 
            maxDataLength + ".  RPC came from " + getHostAddress();
        LOG.warn(error);
        throw new IOException(error);
      }
    }

    /**
     * This method reads in a non-blocking fashion from the channel: 
     * this method is called repeatedly when data is present in the channel; 
     * when it has enough data to process one rpc it processes that rpc.
     * 
     * On the first pass, it processes the connectionHeader, 
     * connectionContext (an outOfBand RPC) and at most one RPC request that 
     * follows that. On future passes it will process at most one RPC request.
     *  
     * Quirky things: dataLengthBuffer (4 bytes) is used to read "hrpc" OR 
     * rpc request length.
     *    
     * @return -1 in case of error, else num bytes read so far
     * @throws IOException - internal error that should not be returned to
     *         client, typically failure to respond to client

View on GitHub (pinned to 2add963021)

Solutions

Reduce the request size on the caller side: chunk, page, or stream the data through HDFS files instead of passing it as RPC arguments
If the large call is legitimate, raise ipc.maximum.data.length on the server (and keep the client's matching setting consistent) and restart the service
Find the offending call from the WARN log (it includes the client address) and check its arguments for accidental unbounded growth
Watch memory when raising the cap: the server allocates buffers per request, so the ceiling is also a DoS guard

Example fix

// before: one enormous RPC argument
proxy.setAcls(path, buildHugeAclList(entries));

// after: chunk into bounded batches under the server cap
for (List<AclEntry> batch : Iterables.partition(entries, 10_000)) {
  proxy.setAcls(path, batch);
}

Defensive patterns

Strategy: validation

Validate before calling

// client-side guard: estimate serialized size before sending a bulk call
byte[] encoded = payload.toByteArray();
int serverMax = conf.getInt("ipc.maximum.data.length", 128 * 1024 * 1024);
if (encoded.length > serverMax) {
  throw new IllegalArgumentException("RPC payload " + encoded.length
      + " exceeds ipc.maximum.data.length=" + serverMax + "; chunk it");
}

Try / catch

try {
  proxy.bulkOp(items);
} catch (RemoteException re) {
  if (re.getMessage().contains("longer than maximum configured RPC length")) {
    // split the batch and retry with smaller chunks
    for (List<Item> part : Iterables.partition(items, partSize)) proxy.bulkOp(part);
  } else { throw re; }
}

Prevention

Bound batch sizes at call sites; stream large data through HDFS files instead of RPC args
Document ipc.maximum.data.length when deploying services with large payloads
Remember the cap doubles as a memory/DoS guard: raise it deliberately, not reflexively

When it happens

Trigger: A client call marshals an argument larger than ipc.maximum.data.length — for example a bulk operation with a huge list, a very large byte[] parameter, or a batched metadata request with hundreds of thousands of entries; server configured with a lower cap than the traffic it serves.

Common situations: Applications calling setXAttrs/addCachePool/listStatus-style APIs with unbounded batch sizes; code that stuffs large payloads (whole files, big JSON) into RPC arguments instead of writing to HDFS; admins who lowered ipc.maximum.data.length for hardening and then a legitimate large request breaks.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5d69a5564f6f558c. Report an issue: GitHub.