{"record":{"id":"389fc9dda4d472b8","repo":"apache/flink","slug":"encoded-string-too-long-memory","errorCode":null,"errorMessage":"encoded string too long: {} memory","messagePattern":"encoded string too long: (.+?) memory","errorType":"exception","errorClass":"UTFDataFormatException","httpStatus":null,"severity":"error","filePath":"flink-core/src/main/java/org/apache/flink/core/memory/AbstractPagedOutputView.java","lineNumber":334,"sourceCode":"    public void writeUTF(String str) throws IOException {\n        int strlen = str.length();\n        int utflen = 0;\n        int c, count = 0;\n\n        /* use charAt instead of copying String to char array */\n        for (int i = 0; i < strlen; i++) {\n            c = str.charAt(i);\n            if ((c >= 0x0001) && (c <= 0x007F)) {\n                utflen++;\n            } else if (c > 0x07FF) {\n                utflen += 3;\n            } else {\n                utflen += 2;\n            }\n        }\n\n        if (utflen > 65535) {\n            throw new UTFDataFormatException(\"encoded string too long: \" + utflen + \" memory\");\n        }\n\n        if (this.utfBuffer == null || this.utfBuffer.length < utflen + 2) {\n            this.utfBuffer = new byte[utflen + 2];\n        }\n        final byte[] bytearr = this.utfBuffer;\n\n        bytearr[count++] = (byte) ((utflen >>> 8) & 0xFF);\n        bytearr[count++] = (byte) (utflen & 0xFF);\n\n        int i;\n        for (i = 0; i < strlen; i++) {\n            c = str.charAt(i);\n            if (!((c >= 0x0001) && (c <= 0x007F))) {\n                break;\n            }\n            bytearr[count++] = (byte) c;\n        }","sourceCodeStart":316,"sourceCodeEnd":352,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-core/src/main/java/org/apache/flink/core/memory/AbstractPagedOutputView.java#L316-L352","documentation":"Thrown as a UTFDataFormatException by AbstractPagedOutputView.writeUTF when the modified-UTF-8 encoding of the string exceeds 65535 bytes. The format prefixes the string with a 2-byte length field (unsigned short, max 65535), so the encoded byte length must fit in 16 bits. This mirrors java.io.DataOutputStream.writeUTF's limitation.","triggerScenarios":"Writing a string via writeUTF whose modified-UTF-8 byte representation is longer than 65535 bytes; long strings with many non-ASCII characters (3 bytes each in modified UTF-8) hit this limit sooner than ASCII-heavy strings.","commonSituations":"Serializing large text fields (e.g. JSON payloads, log lines, descriptions) via writeUTF; strings that are within Java's UTF-16 length but exceed the modified-UTF-8 byte budget.","solutions":["Shorten the string or split it into chunks each under 65535 modified-UTF-8 bytes.","Use a length-prefixed byte serialization (writeInt(length) + write(bytes)) instead of writeUTF for large strings.","If the string is a serialized object, switch to a binary serialization format that does not have the 64KB limit.","Validate string encoded length before writing and handle gracefully."],"exampleFix":"// before\noutputView.writeUTF(veryLongString);\n\n// after — length-prefixed raw bytes for large strings\nbyte[] bytes = veryLongString.getBytes(StandardCharsets.UTF_8);\noutputView.writeInt(bytes.length);\noutputView.write(bytes);","handlingStrategy":"validation","validationCode":"int utflen = computeModifiedUtf8Length(str);\nif (utflen > 65535) throw new UTFDataFormatException(\"String too long for writeUTF: \" + utflen + \" bytes\");","typeGuard":null,"tryCatchPattern":"try {\n    outputView.writeUTF(str);\n} catch (UTFDataFormatException e) {\n    if (e.getMessage().contains(\"too long\")) {\n        // fall back to length-prefixed byte serialization\n        byte[] b = str.getBytes(StandardCharsets.UTF_8);\n        outputView.writeInt(b.length);\n        outputView.write(b);\n    }\n}","preventionTips":["Prefer length-prefixed byte serialization (writeInt + write) for strings that may exceed 64KB.","Validate encoded length before writeUTF for variable-length text fields.","Be aware non-ASCII characters consume 2-3 bytes in modified UTF-8."],"tags":["memory","paged-io","utf-8","size-limit"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}