apache/hadoop · error · IOException

string was too long to write! Expected less than or equal t

Error message

string was too long to write!  Expected less than or equal to {} bytes, but got {} bytes.

What it means

Text.writeString(DataOutput, String, int maxLength) encodes the string to UTF-8 and refuses to write more than maxLength bytes, throwing IOException("string was too long to write! Expected less than or equal to M bytes, but got N bytes."). The limit is on encoded bytes, not characters, and the method returns the encoded length on success.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java:603

    WritableUtils.writeVInt(out, length);
    out.write(bytes.array(), 0, length);
    return length;
  }

  /**
   * @return Write a UTF8 encoded string with a maximum size to out.
   *
   * @param out input out.
   * @param s input s.
   * @param maxLength input maxLength.
   * @throws IOException raised on errors performing I/O.
   */
  public static int writeString(DataOutput out, String s, int maxLength)
      throws IOException {
    ByteBuffer bytes = encode(s);
    int length = bytes.limit();
    if (length > maxLength) {
      throw new IOException("string was too long to write!  Expected " +
          "less than or equal to " + maxLength + " bytes, but got " +
          length + " bytes.");
    }
    WritableUtils.writeVInt(out, length);
    out.write(bytes.array(), 0, length);
    return length;
  }

  ////// states for validateUTF8

  private static final int LEAD_BYTE = 0;

  private static final int TRAIL_BYTE_1 = 1;

  private static final int TRAIL_BYTE = 2;

  /**
   * Check if a byte array contains valid UTF-8.

View on GitHub (pinned to 2add963021)

Solutions

1. Check the encoded size before writing: int len = Text.encode(s).limit(); then reject or truncate deliberately.
2. Raise maxLength on both write and read paths if larger values are legitimate.
3. Test limits with multi-byte content (3-byte CJK, 4-byte emoji), not just ASCII.

Example fix

// before
Text.writeString(out, s, 256);           // throws on long/multibyte strings

// after: fail with an actionable error at the call site
ByteBuffer b = Text.encode(s);
if (b.limit() > MAX_LEN) {
  throw new IllegalArgumentException("value is " + b.limit()
      + " UTF-8 bytes, limit " + MAX_LEN);
}
Text.writeString(out, s, MAX_LEN);

ByteBuffer b = Text.encode(s);
if (b.limit() > maxLength) {
  throw new IllegalArgumentException(
      "String encodes to " + b.limit() + " UTF-8 bytes, limit " + maxLength);
}
Text.writeString(out, s, maxLength);

When it happens

Trigger: Serializing a string whose UTF-8 byte length exceeds the limit the format/reader imposes; strings that look short in characters but expand 3-4x in UTF-8 (CJK, emoji); raising the reader limit without raising the writer's maxLength (or vice versa).

Common situations: Config keys/values, names, or identifiers written into length-bounded slots; internationalized content blowing past limits sized for ASCII; mismatched caps after a limit change.

Related errors

• data was too long to write! Expected less than or equal to
• tried to deserialize {} bytes of data, but maxLength = {}
• parent + " is a file"
• Parent directory doesn't exist: " + parent
• absolutePath + ": No such file or directory."

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7e6450a930493130. Report an issue: GitHub.