apache/flink · error · UTFDataFormatException

Encoded string is too long: {}

Error message

Encoded string is too long: {}

What it means

Thrown by Record's writeUTF implementation, mirroring java.io.DataOutputStream.writeUTF: before writing, it counts the UTF-8 encoded length of the string (1 byte per char in U+0001..U+007F, 2 or 3 bytes otherwise) and rejects strings whose encoding exceeds 65,535 bytes, because the format stores the length in an unsigned 2-byte prefix. This is a UTFDataFormatException, an IOException subtype.

Source

Thrown at flink-core/src/main/java/org/apache/flink/types/Record.java:1743

        public void writeUTF(String str) throws IOException {
            int strlen = str.length();
            int utflen = 0;
            int c;

            /* use charAt instead of copying String to char array */
            for (int i = 0; i < strlen; i++) {
                c = str.charAt(i);
                if ((c >= 0x0001) && (c <= 0x007F)) {
                    utflen++;
                } else if (c > 0x07FF) {
                    utflen += 3;
                } else {
                    utflen += 2;
                }
            }

            if (utflen > 65535) {
                throw new UTFDataFormatException("Encoded string is too long: " + utflen);
            } else if (this.position > this.memory.length - utflen - 2) {
                resize(utflen + 2);
            }

            byte[] bytearr = this.memory;
            int count = this.position;

            bytearr[count++] = (byte) ((utflen >>> 8) & 0xFF);
            bytearr[count++] = (byte) ((utflen >>> 0) & 0xFF);

            int i = 0;
            for (i = 0; i < strlen; i++) {
                c = str.charAt(i);
                if (!((c >= 0x0001) && (c <= 0x007F))) {
                    break;
                }
                bytearr[count++] = (byte) c;
            }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Shorten or truncate the string so its modified-UTF-8 encoding is <= 65535 bytes
2. Move large payloads out of the string field: write them via write(byte[]) after writing a length prefix
3. If you control the schema, split the field or compress the payload before serialization

Example fix

// before
record.setField(0, hugeJsonString); // > 64KB encoded

// after
byte[] payload = compress(hugeJsonString.getBytes(StandardCharsets.UTF_8));
record.setField(0, Base64.getEncoder().encodeToString(payload));

static boolean fitsModifiedUtf8(String s) {
    long utflen = 0;
    for (int i = 0; i < s.length(); i++) {
        char c = s.charAt(i);
        utflen += (c >= 0x0001 && c <= 0x007F) ? 1 : (c > 0x07FF ? 3 : 2);
    }
    return utflen <= 65535;
}

catch (UTFDataFormatException e) { // truncate or offload the payload }

When it happens

Trigger: Calling Record's writeUTF (or serializing a Record containing a very large String field) where the modified-UTF-8 encoding of the string exceeds 65,535 bytes — typically long strings, or strings with many non-Latin characters that encode at 2–3 bytes each.

Common situations: Packing payloads, JSON blobs, or stack traces into a Record string field; migrating data that previously used a different transport without the 64KB limit; strings that pass length checks in chars but blow the limit in encoded bytes due to non-ASCII content.

Related errors

• Byte array does not provide enough space to store requested
• Could not skip {} bytes.
• Could not write {} bytes since the buffer is full.
• Invalid version %d
• Serializing the source elements failed: {e.getMessage()}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/2fdc5c557aaf3802. Report an issue: GitHub.