apache/hadoop · error · RuntimeException

string too long!

Error message

string too long!

What it means

org.apache.hadoop.io.UTF8 is the legacy UTF-8 Writable whose length field is an unsigned 16-bit short. set(String) pre-truncates input longer than 0xffff/3 = 21845 characters (with a WARN) and then encodes; if the encoded byte length still exceeds 0xffff it throws RuntimeException("string too long!"). Because characters can take up to 3 bytes, even pre-truncated strings of CJK content can exceed the 64 KB byte bound.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/UTF8.java:105

  /** @return The number of bytes in the encoded string. */
  public int getLength() {
    return length;
  }

  /**
   * Set to contain the contents of a string.
   * @param string input string.
   */
  public void set(String string) {
    if (string.length() > 0xffff/3) {             // maybe too long
      LOG.warn("truncating long string: " + string.length()
               + " chars, starting with " + string.substring(0, 20));
      string = string.substring(0, 0xffff/3);
    }

    length = utf8Length(string);                  // compute length
    if (length > 0xffff)                          // double-check length
      throw new RuntimeException("string too long!");

    if (bytes == null || length > bytes.length)   // grow buffer
      bytes = new byte[length];

    try {                                         // avoid sync'd allocations
      DataOutputBuffer obuf = OBUF_FACTORY.get();
      obuf.reset();
      writeChars(obuf, string, 0, string.length());
      System.arraycopy(obuf.getData(), 0, bytes, 0, length);
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
  }

  /**
   * Set to contain the contents of a string.
   * @param other input other.
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Switch the field to org.apache.hadoop.io.Text, which stores the length as a VInt (up to ~2 GB) — Text is the supported replacement for UTF8.
2. If UTF8 must stay, cap the encoded size yourself before calling set(): ensure UTF8.utf8Length(s) <= 0xffff, truncating by bytes.
3. Redesign the storage so unbounded strings are not forced through a 16-bit-length container.

Example fix

// before
UTF8 u = new UTF8();
u.set(hugeString);                       // RuntimeException: string too long!

// after
Text t = new Text(hugeString);           // VInt length, no 64 KB bound

static boolean fitsLegacyUtf8(String s) {
  return UTF8.utf8Length(s) <= 0xffff;  // 16-bit length field bound
}

// use before storing into a UTF8 field
if (!fitsLegacyUtf8(s)) {
  s = s.substring(0, Math.min(s.length(), 20000));  // byte-safe for CJK too
}

When it happens

Trigger: Calling UTF8.set(String) with a string whose UTF-8 encoding exceeds 65535 bytes — most easily reached with 3-byte-per-char content at or near the 21845-char pre-truncation point; long URLs, JSON blobs, or messages stored in legacy UTF8-typed fields.

Common situations: Legacy MapFile/UTF8-based schemas receiving modern long strings; East-Asian content where character-count guards pass but byte limits fail; migrating old Writable types that were sized for ASCII identifiers.

Related errors

• tried to deserialize {} bytes of data! newLength must be no
• tried to deserialize {} bytes of data, but maxLength = {}
• data was too long to write! Expected less than or equal to
• string was too long to write! Expected less than or equal t
• Invalid UTF8 at {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f6ca6a480b1eda3d. Report an issue: GitHub.