apache/flink · error · UTFDataFormatException

encoded string too long

Error message

encoded string too long: {} memory

What it means

Thrown as a UTFDataFormatException by AbstractPagedOutputView.writeUTF when the modified-UTF-8 encoding of the string exceeds 65535 bytes. The format prefixes the string with a 2-byte length field (unsigned short, max 65535), so the encoded byte length must fit in 16 bits. This mirrors java.io.DataOutputStream.writeUTF's limitation.

Solutions

1. Shorten the string or split it into chunks each under 65535 modified-UTF-8 bytes.
2. Use a length-prefixed byte serialization (writeInt(length) + write(bytes)) instead of writeUTF for large strings.
3. If the string is a serialized object, switch to a binary serialization format that does not have the 64KB limit.
4. Validate string encoded length before writing and handle gracefully.

Example fix

// before
outputView.writeUTF(veryLongString);

// after — length-prefixed raw bytes for large strings
byte[] bytes = veryLongString.getBytes(StandardCharsets.UTF_8);
outputView.writeInt(bytes.length);
outputView.write(bytes);

int utflen = computeModifiedUtf8Length(str);
if (utflen > 65535) throw new UTFDataFormatException("String too long for writeUTF: " + utflen + " bytes");

try {
    outputView.writeUTF(str);
} catch (UTFDataFormatException e) {
    if (e.getMessage().contains("too long")) {
        // fall back to length-prefixed byte serialization
        byte[] b = str.getBytes(StandardCharsets.UTF_8);
        outputView.writeInt(b.length);
        outputView.write(b);
    }
}

When it happens

Trigger: Writing a string via writeUTF whose modified-UTF-8 byte representation is longer than 65535 bytes; long strings with many non-ASCII characters (3 bytes each in modified UTF-8) hit this limit sooner than ASCII-heavy strings.

Common situations: Serializing large text fields (e.g. JSON payloads, log lines, descriptions) via writeUTF; strings that are within Java's UTF-16 length but exceed the modified-UTF-8 byte budget.

Related errors

• Encoded string reached maximum length
• malformed input around byte
• malformed input: partial character at end
• Encoded string is too long
• Initial Segment may not be null

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/389fc9dda4d472b8. Report an issue: GitHub.

Appendix: source

Thrown at flink-core/src/main/java/org/apache/flink/core/memory/AbstractPagedOutputView.java:334

    public void writeUTF(String str) throws IOException {
        int strlen = str.length();
        int utflen = 0;
        int c, count = 0;

        /* use charAt instead of copying String to char array */
        for (int i = 0; i < strlen; i++) {
            c = str.charAt(i);
            if ((c >= 0x0001) && (c <= 0x007F)) {
                utflen++;
            } else if (c > 0x07FF) {
                utflen += 3;
            } else {
                utflen += 2;
            }
        }

        if (utflen > 65535) {
            throw new UTFDataFormatException("encoded string too long: " + utflen + " memory");
        }

        if (this.utfBuffer == null || this.utfBuffer.length < utflen + 2) {
            this.utfBuffer = new byte[utflen + 2];
        }
        final byte[] bytearr = this.utfBuffer;

        bytearr[count++] = (byte) ((utflen >>> 8) & 0xFF);
        bytearr[count++] = (byte) (utflen & 0xFF);

        int i;
        for (i = 0; i < strlen; i++) {
            c = str.charAt(i);
            if (!((c >= 0x0001) && (c <= 0x007F))) {
                break;
            }
            bytearr[count++] = (byte) c;
        }

View on GitHub (pinned to 2f3c205e92)