apache/iceberg · error · UTFDataFormatException

Encoded string reached maximum length: {utflen}

Error message

Encoded string reached maximum length: {utflen}

What it means

SerializerHelper.writeLongUTF encodes a string as modified UTF-8 and tracks the running byte length. If the encoded length would exceed Integer.MAX_VALUE during the per-character accounting loop, it throws UTFDataFormatException immediately, since the length cannot be represented. This is a hard limit of the DataOutput-style encoding used for Flink serializer compatibility.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/util/SerializerHelper.java:65

   *
   * <p>See * <a href="https://issues.apache.org/jira/browse/FLINK-34228">FLINK-34228</a> * <a
   * href="https://github.com/apache/flink/pull/24191">https://github.com/apache/flink/pull/24191</a>
   *
   * @param out the output stream to write the string to.
   * @param str the string value to be written.
   */
  public static void writeLongUTF(DataOutputView out, String str) throws IOException {
    int strlen = str.length();
    long utflen = 0;
    int ch;

    /* use charAt instead of copying String to char array */
    for (int i = 0; i < strlen; i++) {
      ch = str.charAt(i);
      utflen += getUTFBytesSize(ch);

      if (utflen > Integer.MAX_VALUE) {
        throw new UTFDataFormatException("Encoded string reached maximum length: " + utflen);
      }
    }

    if (utflen > Integer.MAX_VALUE - 4) {
      throw new UTFDataFormatException("Encoded string is too long: " + utflen);
    }

    out.writeInt((int) utflen);
    writeUTFBytes(out, str, (int) utflen);
  }

  /**
   * Similar to {@link DataInputDeserializer#readUTF()}. Except this supports larger payloads which
   * is up to max integer value.
   *
   * <p>Note: This method can be removed when the method which does similar thing within the {@link
   * DataOutputSerializer} already which does the same thing, so use that one instead once that is
   * released on Flink version 1.20.

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Reduce the string size below Integer.MAX_VALUE encoded bytes before writing
2. Split the payload into chunks and write multiple strings
3. Use a different transport (e.g. binary byte arrays or external storage) for >2GB payloads

Example fix

// before
helper.writeLongUTF(hugePayload); // throws if > Integer.MAX_VALUE bytes
// after
if (getUTFBytesSize(hugePayload) < Integer.MAX_VALUE) {
  helper.writeLongUTF(hugePayload);
} else {
  writeChunked(hugePayload);
}

int size = SerializerHelper.getUTFBytesSize(str);
if (size >= Integer.MAX_VALUE) {
  throw new IllegalArgumentException("string exceeds encoder limit: " + size);
}
helper.writeLongUTF(str);

try {
  helper.writeLongUTF(str);
} catch (UTFDataFormatException e) {
  // fall back to chunked storage or external reference
  writeExternalRef(str);
}

When it happens

Trigger: Calling writeLongUTF with a string whose UTF-8 encoding exceeds 2^31-1 bytes (~2GB).

Common situations: Serializing extremely large strings such as huge JSON payloads or binary data smuggled through String columns across Flink checkpoints/state.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Encoded string is too long: {utflen}
• malformed input: partial character at end
• malformed input around byte {count}
• malformed input around byte
• Encoded string reached maximum length:

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/815599b52ed736ea. Report an issue: GitHub.