apache/hadoop · error · EOFException

String length ${length} exceeds the limit of ${maxLength}

Error message

String length ${length} exceeds the limit of ${maxLength}

What it means

Guard inside Utils.readString(DataInput in, int maxLength): after reading the VInt length prefix, if maxLength >= 0 and length > maxLength the method throws EOFException("String length ... exceeds the limit of ..."). The bound exists to stop a corrupt or hostile length prefix (values near 2^31) from triggering a huge byte[] allocation and OOM; it converts a memory-exhaustion failure into a fast, descriptive one.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Utils.java:305

   * @param in The input stream.
   * @param maxLength The largest permitted encoded length in bytes, negative for no limit.
   * @return The string or null.
   * @throws EOFException input data length exceeds {@code maxLength}.
   * @throws IOException IO failure.
   * @throws NegativeArraySizeException string length was minus two or less.
   */
  public static String readString(DataInput in, int maxLength)
      throws IOException {
    int length = readVInt(in);
    if (length == -1) {
      return null;
    }
    if (length < 0) {
      throw new NegativeArraySizeException("Corrupted data: negative string length "
          + length);
    }
    if (maxLength >= 0 && length > maxLength) {
      throw new EOFException("String length " + length
          + " exceeds the limit of " + maxLength);
    }
    byte[] buffer = new byte[length];
    in.readFully(buffer);
    return Text.decode(buffer);
  }

  /**
   * A generic Version class. We suggest applications built on top of TFile use
   * this class to maintain version information in their meta blocks.
   * 
   * A version number consists of a major version and a minor version. The
   * suggested usage of major and minor version number is to increment major
   * version number when the new storage format is not backward compatible, and
   * increment the minor version otherwise.
   */
  public static final class Version implements Comparable<Version> {
    private final short major;

View on GitHub (pinned to 2add963021)

Solutions

1. Compare the length in the message against your maxLength: if the input is known-good, the caller's limit is wrong — raise it to the format's true maximum.
2. If the reported length is absurdly large, treat the input as corrupt (same remediation as the negative-length sibling) and regenerate the file.
3. When you control both ends, derive the limit from the format spec and use the same constant for write and read so they cannot diverge.

Example fix

// before: limit smaller than legal values
String s = Utils.readString(in, 1024);

// after: limit derived from the format's true maximum string size
String s = Utils.readString(in, MAX_KEY_LENGTH /* e.g. 64 * 1024 */);

try {
  String s = Utils.readString(in, maxLength);
} catch (EOFException e) {
  // limit exceeded (or truncated stream): decide corrupt vs misconfigured limit
  // from the reported length before any retry
}

When it happens

Trigger: Calling readString with a maxLength bound where the stream's length prefix exceeds it — genuinely corrupt data with an inflated length field, a mismatch between the limit the writer assumed and the one the reader enforces, or a crafted/fuzzed input file.

Common situations: After upgrading Hadoop, callers now pass a maxLength to data written by older code with longer legitimate strings; an arbitrarily small maxLength chosen by the caller; corrupted TFile metadata where the length field is garbage but positive.

Related errors

• Key length out of range: {klen}
• Cannot find matching key in block.
• First key entry size out of range: {size}
• First key length out of range: {firstKeyLength}
• Index entry size out of range: {size}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/cf820b72b096fbf2. Report an issue: GitHub.