apache/hadoop · error · IOException

data was too long to write! Expected less than or equal to

Error message

data was too long to write!  Expected less than or equal to {} bytes, but got {} bytes.

What it means

Text.write(DataOutput, int maxLength) enforces a caller-supplied byte cap before serializing: if the Text's current byte length exceeds maxLength it throws IOException("data was too long to write! Expected less than or equal to M bytes, but got N bytes."). It lets fixed-boundary formats reject oversized fields deterministically instead of writing a record the reader will refuse.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java:403

    in.readFully(bytes, 0, len);
    length = len;
    textLength = -1;
  }

  /**
   * Serialize. Write this object to out length uses zero-compressed encoding.
   *
   * @see Writable#write(DataOutput)
   */
  @Override
  public void write(DataOutput out) throws IOException {
    WritableUtils.writeVInt(out, length);
    out.write(bytes, 0, length);
  }

  public void write(DataOutput out, int maxLength) throws IOException {
    if (length > maxLength) {
      throw new IOException("data was too long to write!  Expected " +
          "less than or equal to " + maxLength + " bytes, but got " +
          length + " bytes.");
    }
    WritableUtils.writeVInt(out, length);
    out.write(bytes, 0, length);
  }

  /**
   * Returns true iff <code>o</code> is a Text with the same length and same
   * contents.
   */
  @Override
  public boolean equals(Object o) {
    if (o instanceof Text)
      return super.equals(o);
    return false;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Check text.getLength() against the cap up front and truncate to valid UTF-8 boundaries or reject with an actionable error.
2. Raise the cap on both writer and reader if larger values are legitimate for the format.
3. Validate input strings at ingestion (before they reach serialization) against the same limit.

Example fix

// before
text.write(out, MAX_FIELD_BYTES);        // throws when text is longer

// after: enforce the policy before serializing
if (text.getLength() > MAX_FIELD_BYTES) {
  throw new IllegalArgumentException("value exceeds " + MAX_FIELD_BYTES
      + " bytes: " + text);
}
text.write(out, MAX_FIELD_BYTES);

if (text.getLength() > maxLength) {
  throw new IllegalArgumentException(
      "Text is " + text.getLength() + " bytes, limit " + maxLength);
}
text.write(out, maxLength);

When it happens

Trigger: Writing a Text into a length-bounded slot (file-format field, fixed-size record area, protocol field with a documented maximum) where the value's UTF-8 byte length exceeds the cap; limits tuned in characters but enforced in bytes with multi-byte content.

Common situations: User-supplied strings (names, identifiers, attributes) exceeding format limits; limits documented in characters but enforced on UTF-8 bytes; writer and reader disagreeing on the cap.

Related errors

• tried to deserialize {} bytes of data, but maxLength = {}
• string was too long to write! Expected less than or equal t
• Exception while get content summary
• f.toString()
• Class {} already registered but maps to {} and not {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4e37d8f042eb830f. Report an issue: GitHub.