apache/iceberg · error · UTFDataFormatException

malformed input: partial character at end

Error message

malformed input: partial character at end

What it means

readLongUTF validates that every declared 2-byte (110xxxxx) character fits entirely within the declared UTF length; when the final byte boundary cuts a character in half (count > utflen) it throws UTFDataFormatException('malformed input: partial character at end'). This means the incoming byte stream is truncated or its declared length does not match its content.

Source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/util/SerializerHelper.java:134

      switch (ch >> 4) {
        case 0:
        case 1:
        case 2:
        case 3:
        case 4:
        case 5:
        case 6:
        case 7:
          /* 0xxxxxxx */
          count++;
          chararr[chararrCount++] = (char) ch;
          break;
        case 12:
        case 13:
          /* 110x xxxx 10xx xxxx */
          count += 2;
          if (count > utflen) {
            throw new UTFDataFormatException("malformed input: partial character at end");
          }
          char2 = bytearr[count - 1];
          if ((char2 & 0xC0) != 0x80) {
            throw new UTFDataFormatException("malformed input around byte " + count);
          }
          chararr[chararrCount++] = (char) (((ch & 0x1F) << 6) | (char2 & 0x3F));
          break;
        case 14:
          /* 1110 xxxx 10xx xxxx 10xx xxxx */
          count += 3;
          if (count > utflen) {
            throw new UTFDataFormatException("malformed input: partial character at end");
          }
          char2 = bytearr[count - 2];
          char3 = bytearr[count - 1];
          if (((char2 & 0xC0) != 0x80) || ((char3 & 0xC0) != 0x80)) {
            throw new UTFDataFormatException("malformed input around byte " + (count - 1));
          }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Validate the producer wrote the correct length header and the full byte array (check writer/reader version parity)
2. Re-read or re-fetch the payload — check for truncation in the transport (Kafka message size limits, socket timeouts)
3. Catch UTFDataFormatException on read and log the byte offsets to isolate the corrupted record, then re-serialize the affected data

Example fix

// before
String s = SerializerHelper.readLongUTF(in); // throws on truncated input
// after
try {
  String s = SerializerHelper.readLongUTF(in);
} catch (UTFDataFormatException e) {
  LOG.warn("Truncated/corrupted UTF payload, re-fetching record", e);
  s = fallbackRead();
}

// before reading: verify buffer holds utflen bytes after the int header
if (buf.position() + declaredLen > buf.limit()) {
  throw new EOFException("Declared UTF length exceeds available bytes");
}

try {
  String s = SerializerHelper.readLongUTF(in);
} catch (UTFDataFormatException e) {
  if (e.getMessage().contains("partial character at end") || e.getMessage().contains("malformed input")) {
    log.warn("Truncated/corrupt UTF record at offset {}", bytesConsumed, e);
    s = null; // re-fetch or skip record
  } else throw e;
}

When it happens

Trigger: Deserializing a payload where the declared int length prefix is larger than the actual encoded bytes, or the stream was truncated mid-character (network cut, partial write, corrupted file).

Common situations: Kafka/serialization payloads cut off by buffer limits; mismatched writer/reader versions producing inconsistent length headers; corruption when copying serialized snapshots.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• malformed input: partial character at end
• malformed input: partial character at end
• malformed input around byte
• Encoded string reached maximum length: {utflen}
• Encoded string is too long: {utflen}

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/166617735c5fb193. Report an issue: GitHub.