apache/iceberg · error · UTFDataFormatException

Encoded string reached maximum length:

Error message

Encoded string reached maximum length: 

What it means

SerializerHelper.writeLongUTF computes the UTF-8 encoded length of the string in a long accumulator while iterating characters; if the running length exceeds Integer.MAX_VALUE it throws UTFDataFormatException('Encoded string reached maximum length'). This guards the modified UTF-style writer, which ultimately writes the length as an int.

Source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/util/SerializerHelper.java:65

   *
   * <p>See * <a href="https://issues.apache.org/jira/browse/FLINK-34228">FLINK-34228</a> * <a
   * href="https://github.com/apache/flink/pull/24191">https://github.com/apache/flink/pull/24191</a>
   *
   * @param out the output stream to write the string to.
   * @param str the string value to be written.
   */
  public static void writeLongUTF(DataOutputView out, String str) throws IOException {
    int strlen = str.length();
    long utflen = 0;
    int ch;

    /* use charAt instead of copying String to char array */
    for (int i = 0; i < strlen; i++) {
      ch = str.charAt(i);
      utflen += getUTFBytesSize(ch);

      if (utflen > Integer.MAX_VALUE) {
        throw new UTFDataFormatException("Encoded string reached maximum length: " + utflen);
      }
    }

    if (utflen > Integer.MAX_VALUE - 4) {
      throw new UTFDataFormatException("Encoded string is too long: " + utflen);
    }

    out.writeInt((int) utflen);
    writeUTFBytes(out, str, (int) utflen);
  }

  /**
   * Similar to {@link DataInputDeserializer#readUTF()}. Except this supports larger payloads which
   * is up to max integer value.
   *
   * <p>Note: This method can be removed when the method which does similar thing within the {@link
   * DataOutputSerializer} already which does the same thing, so use that one instead once that is
   * released on Flink version 1.20.

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Split or chunk the payload across multiple records/fields before serialization
2. Compress the payload (e.g. gzip + base64) so encoded size stays under the limit
3. Store large payloads externally (object storage) and serialize only a reference

Example fix

// before
String huge = buildGiantJson(); // > 2GB encoded
helper.writeLongUTF(out, huge);
// after
byte[] compressed = gzip(huge.getBytes(StandardCharsets.UTF_8));
out.writeInt(compressed.length);
out.write(compressed);

long utfSize = 0;
for (int i = 0; i < str.length(); i++) { utfSize += utfByteLen(str.charAt(i)); }
if (utfSize > Integer.MAX_VALUE) throw new IllegalArgumentException("String exceeds encodable size");
// or compute with StandardCharsets.UTF_8 encoder before writing

static boolean isEncodable(String s) {
  return s != null && s.getBytes(StandardCharsets.UTF_8).length < Integer.MAX_VALUE;
}

try { helper.writeLongUTF(out, str); }
catch (UTFDataFormatException e) {
  if (e.getMessage().contains("reached maximum length") || e.getMessage().contains("too long")) { chunkAndWrite(out, str); }
  else throw e;
}

When it happens

Trigger: Serializing a single String whose UTF-8 encoding exceeds ~2.1 GB (Integer.MAX_VALUE bytes) via writeLongUTF — practically only with enormous string values in Kafka/serializer payloads.

Common situations: Jobs that pack gigantic blobs (e.g. huge JSON, base64 payloads) into a single string column/attribute and then pass them through the custom UTF serializer.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• Encoded string is too long:
• Encoded string reached maximum length: {utflen}
• Encoded string is too long: {utflen}
• malformed input: partial character at end
• malformed input around byte {count}

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/262f691997e9fb09. Report an issue: GitHub.