apache/hadoop · error · IOException

Illegal buffer length " + len

Error message

Illegal buffer length " + len

What it means

Thrown while Java-deserializing a RawPathHandle: after defaultReadObject, the stream's next int is the handle byte length, and it must be in [0, 1<<20] (1 MiB, RawPathHandle.MAX_SIZE). A length outside that window means the stream is corrupted, truncated, version-skewed between writer and reader, or deliberately hostile (the cap is an OOM guard against crafted length fields). Deserialization is aborted with IOException.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawPathHandle.java:108

  private void writeObject(ObjectOutputStream out) throws IOException {
    out.defaultWriteObject();
    out.writeInt(fd.remaining());
    if (fd.hasArray()) {
      out.write(fd.array(), fd.position(), fd.remaining());
    } else {
      byte[] x = new byte[fd.remaining()];
      fd.slice().get(x);
      out.write(x);
    }
  }

  private void readObject(ObjectInputStream in)
      throws IOException, ClassNotFoundException {
    in.defaultReadObject();
    int len = in.readInt();
    if (len < 0 || len > MAX_SIZE) {
      throw new IOException("Illegal buffer length " + len);
    }
    byte[] x = new byte[len];
    in.readFully(x);
    fd = ByteBuffer.wrap(x);
  }

  private void readObjectNoData() throws ObjectStreamException {
    throw new InvalidObjectException("Stream data required");
  }

}

View on GitHub (pinned to 2add963021)

Solutions

1. Discard the stored/queued handle and re-create it from the source FileSystem rather than repairing the bytes
2. Verify end-to-end integrity: checksum the serialized blob, or confirm both ends run the same Hadoop version
3. Never ObjectInputStream.readObject() on untrusted data; wrap handles in a length- and checksum-validated envelope

Example fix

// before: trusting a raw byte stream
try (ObjectInputStream in = new ObjectInputStream(bais)) {
  RawPathHandle h = (RawPathHandle) in.readObject();
}

// after: rebuild from the authoritative source
PathHandle h = fs.getPathHandle(fs.getFileStatus(p));

// producer side: refuse to emit oversized handles
ByteBuffer b = handle.bytes();
if (b.remaining() > RawPathHandle.MAX_SIZE) {
  throw new IOException("Handle exceeds " + RawPathHandle.MAX_SIZE + " bytes");
}

try {
  Object o = in.readObject();
} catch (IOException e) {
  // "Illegal buffer length N" -> corrupt/version-skewed stream
  PathHandle h = fs.getPathHandle(fs.getFileStatus(p)); // rebuild from source
}

When it happens

Trigger: ObjectInputStream.readObject() on bytes that were not produced by RawPathHandle.writeObject (truncation mid-stream, garbage payload, an int misaligned by a class-shape change), or an attacker-controlled stream declaring length > 1048576 or negative.

Common situations: Passing serialized PathHandles between client and server versions whose serialization layout differs; storing handle blobs in a queue/db where the payload got mangled; accepting serialized objects over an unauthenticated endpoint.

Related errors

• Stream data required
• Can't read FileStatusProto with negative size of ${size}
• No URI in deserialized Path
• PathHandle only available for files
• Wrong FileSystem: " + stat.getPath()

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7fb99b6e3a96e4da. Report an issue: GitHub.