apache/flink · error · IllegalArgumentException

CharSequence is too long.

Error message

CharSequence is too long.

What it means

Thrown by StringValue.writeString(CharSequence, DataOutput) when the character sequence length overflows the int used for the length header. The wire format stores strlen + 1 (zero is reserved for null), so a sequence of length Integer.MAX_VALUE makes lenToWrite wrap to a negative int. This is a hard limit of Flink's variable-length string encoding: a single CharSequence of ~2^31 chars cannot be serialized with this method.

Source

Thrown at flink-core/src/main/java/org/apache/flink/types/StringValue.java:806

                    c |= (curr & 0x7f) << shift;
                    shift += 7;
                }
                c |= curr << shift;
            }
            data[i] = (char) c;
        }

        return new String(data, 0, len);
    }

    public static final void writeString(CharSequence cs, DataOutput out) throws IOException {
        if (cs != null) {
            int strlen = cs.length();

            // the length we write is offset by one, because a length of zero indicates a null value
            int lenToWrite = strlen + 1;
            if (lenToWrite < 0) {
                throw new IllegalArgumentException("CharSequence is too long.");
            }

            // string is prefixed by it's variable length encoded size, which can take 1-5 bytes.
            if (lenToWrite < HIGH_BIT) {
                out.write((byte) lenToWrite);
            } else if (lenToWrite < HIGH_BIT14) {
                out.write((lenToWrite | HIGH_BIT));
                out.write((lenToWrite >>> 7));
            } else if (lenToWrite < HIGH_BIT21) {
                out.write(lenToWrite | HIGH_BIT);
                out.write((lenToWrite >>> 7) | HIGH_BIT);
                out.write((lenToWrite >>> 14));
            } else if (lenToWrite < HIGH_BIT28) {
                out.write(lenToWrite | HIGH_BIT);
                out.write((lenToWrite >>> 7) | HIGH_BIT);
                out.write((lenToWrite >>> 14) | HIGH_BIT);
                out.write((lenToWrite >>> 21));
            } else {

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Investigate why a single field is near 2^31 characters; it is almost always a bug (unbounded concatenation or wrong field boundaries) rather than legitimate data.
2. Split the payload into multiple records/fields or stream it through a different representation (byte[], or a file/source split) instead of one CharSequence.
3. Pre-check cs.length() < Integer.MAX_VALUE before calling writeString and fail with a domain-specific error naming the offending field.
4. If huge single values are genuinely required, serialize with a format without the int-length limit rather than this legacy encoding.

Example fix

// before
StringValue.writeString(hugeCharSequence, dataOutput); // may overflow length header

// after
if (hugeCharSequence.length() >= Integer.MAX_VALUE) {
    throw new IllegalArgumentException(
        "Field too large for StringValue encoding: " + hugeCharSequence.length() + " chars");
}
StringValue.writeString(hugeCharSequence, dataOutput);

if (cs == null || cs.length() >= Integer.MAX_VALUE) {
    throw new IllegalArgumentException(
        "CharSequence too long for StringValue encoding: " + (cs == null ? -1 : cs.length()));
}

try {
    StringValue.writeString(cs, out);
} catch (IllegalArgumentException e) {
    if (!"CharSequence is too long.".equals(e.getMessage())) throw e;
    // split payload or fail with domain-specific error
}

When it happens

Trigger: Calling StringValue.writeString(cs, out) (directly or via StringValue.write()/TypeSerializer paths that delegate to it) with a CharSequence whose length() >= Integer.MAX_VALUE; lenToWrite = strlen + 1 becomes negative and the guard at StringValue.java:805 fires.

Common situations: Building giant in-memory strings/buffers (e.g. concatenating large datasets or base64 blobs into one field), reading a huge file into a single String, or a buggy upstream producing an unbounded CharSequence. Realistically hit only with multi-GB character arrays (2GB of chars = 4GB heap).

Related errors

• Error during Java serialization.
• Error during Java deserialization.
• The Kryo Output still contains data from a previous serializ
• Could not clone serializer instance of class {className}
• Unrecognized type. This method is deprecated and might not w

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/ba133ebbb2f577bc. Report an issue: GitHub.