apache/flink · error · UTFDataFormatException

encoded string too long: {} memory

Error message

encoded string too long: {} memory

What it means

Thrown as a UTFDataFormatException by AbstractPagedOutputView.writeUTF when the modified-UTF-8 encoding of the string exceeds 65535 bytes. The format prefixes the string with a 2-byte length field (unsigned short, max 65535), so the encoded byte length must fit in 16 bits. This mirrors java.io.DataOutputStream.writeUTF's limitation.

Source

Thrown at flink-core/src/main/java/org/apache/flink/core/memory/AbstractPagedOutputView.java:334

    public void writeUTF(String str) throws IOException {
        int strlen = str.length();
        int utflen = 0;
        int c, count = 0;

        /* use charAt instead of copying String to char array */
        for (int i = 0; i < strlen; i++) {
            c = str.charAt(i);
            if ((c >= 0x0001) && (c <= 0x007F)) {
                utflen++;
            } else if (c > 0x07FF) {
                utflen += 3;
            } else {
                utflen += 2;
            }
        }

        if (utflen > 65535) {
            throw new UTFDataFormatException("encoded string too long: " + utflen + " memory");
        }

        if (this.utfBuffer == null || this.utfBuffer.length < utflen + 2) {
            this.utfBuffer = new byte[utflen + 2];
        }
        final byte[] bytearr = this.utfBuffer;

        bytearr[count++] = (byte) ((utflen >>> 8) & 0xFF);
        bytearr[count++] = (byte) (utflen & 0xFF);

        int i;
        for (i = 0; i < strlen; i++) {
            c = str.charAt(i);
            if (!((c >= 0x0001) && (c <= 0x007F))) {
                break;
            }
            bytearr[count++] = (byte) c;
        }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Shorten the string or split it into chunks each under 65535 modified-UTF-8 bytes.
2. Use a length-prefixed byte serialization (writeInt(length) + write(bytes)) instead of writeUTF for large strings.
3. If the string is a serialized object, switch to a binary serialization format that does not have the 64KB limit.
4. Validate string encoded length before writing and handle gracefully.

Example fix

// before
outputView.writeUTF(veryLongString);

// after — length-prefixed raw bytes for large strings
byte[] bytes = veryLongString.getBytes(StandardCharsets.UTF_8);
outputView.writeInt(bytes.length);
outputView.write(bytes);

int utflen = computeModifiedUtf8Length(str);
if (utflen > 65535) throw new UTFDataFormatException("String too long for writeUTF: " + utflen + " bytes");

try {
    outputView.writeUTF(str);
} catch (UTFDataFormatException e) {
    if (e.getMessage().contains("too long")) {
        // fall back to length-prefixed byte serialization
        byte[] b = str.getBytes(StandardCharsets.UTF_8);
        outputView.writeInt(b.length);
        outputView.write(b);
    }
}

When it happens

Trigger: Writing a string via writeUTF whose modified-UTF-8 byte representation is longer than 65535 bytes; long strings with many non-ASCII characters (3 bytes each in modified UTF-8) hit this limit sooner than ASCII-heavy strings.

Common situations: Serializing large text fields (e.g. JSON payloads, log lines, descriptions) via writeUTF; strings that are within Java's UTF-16 length but exceed the modified-UTF-8 byte budget.

Related errors

• malformed input: partial character at end
• malformed input around byte {}
• Encoded string reached maximum length: {utflen}
• There is no enough data left in the DataInputView.
• Initial Segment may not be null

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/389fc9dda4d472b8. Report an issue: GitHub.