apache/beam · error · CoderException

Invalid encoded string length

Error message

Invalid encoded string length: {}

What it means

StringUtf8Coder.decode reads a VarInt length prefix followed by that many UTF-8 bytes. A negative decoded length means the stream is corrupt or misaligned — no valid string has negative length, so the coder refuses and throws CoderException. This guards against decoding garbage or decoding at the wrong stream offset.

Solutions

1. Ensure the stream was written by StringUtf8Coder.encode(), including its VarInt length prefix.
2. Verify the decode offset: start at the exact position where encoding began.
3. Regenerate or re-serialize data with the same Beam/coder version that wrote it.
4. Catch CoderException and log/dump the offending bytes to diagnose corruption.

Example fix

// before
String s = StringUtf8Coder.of().decode(rawUtf8Stream);
// after
ByteArrayOutputStream bos = new ByteArrayOutputStream();
StringUtf8Coder.of().encode(myString, bos);
String s = StringUtf8Coder.of().decode(new ByteArrayInputStream(bos.toByteArray()));

in.mark(1);
int first = in.read();
in.reset();
if (first == -1) throw new IllegalStateException("Empty stream, not a StringUtf8-encoded value");

static boolean looksLikeLengthPrefixed(byte[] data) {
  return data != null && data.length >= 1;
}

try {
  String s = StringUtf8Coder.of().decode(inStream);
} catch (CoderException e) {
  log.error("Corrupt or misaligned string encoding", e);
  throw new DataCorruptionException(e);
}

When it happens

Trigger: Decoding a byte stream not produced by StringUtf8Coder (e.g. raw UTF-8 bytes without the VarInt length prefix); decoding from the wrong offset in a concatenated or multiplexed stream; truncated or bit-rotted data where VarInt bytes decode to a negative value.

Common situations: Mixing coder versions between pipeline stages; replaying old serialized data written in a different format; hand-crafted test byte arrays missing the length prefix; seeking into the middle of an encoded stream.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• EOF encountered decoding a ValueKind
• error when decoding a textual integer
• Invalid encoded SolrDocument length
• Unable to decode constant members from payload for…
• Unknown ValueKind number

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2ca7b92150279092. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/coders/StringUtf8Coder.java:57

  public static StringUtf8Coder of() {
    return INSTANCE;
  }

  /////////////////////////////////////////////////////////////////////////////

  private static final StringUtf8Coder INSTANCE = new StringUtf8Coder();
  private static final TypeDescriptor<String> TYPE_DESCRIPTOR = new TypeDescriptor<String>() {};

  private static void writeString(String value, OutputStream dos) throws IOException {
    byte[] bytes = value.getBytes(StandardCharsets.UTF_8);
    VarInt.encode(bytes.length, dos);
    dos.write(bytes);
  }

  private static String readString(InputStream dis) throws IOException {
    int len = VarInt.decodeInt(dis);
    if (len < 0) {
      throw new CoderException("Invalid encoded string length: " + len);
    }
    byte[] bytes = new byte[len];
    ByteStreams.readFully(dis, bytes);
    return new String(bytes, StandardCharsets.UTF_8);
  }

  private StringUtf8Coder() {}

  @Override
  public void encode(String value, OutputStream outStream) throws IOException {
    encode(value, outStream, Context.NESTED);
  }

  @Override
  public void encode(String value, OutputStream outStream, Context context) throws IOException {
    if (value == null) {
      throw new CoderException("cannot encode a null String");
    }

View on GitHub (pinned to 12126d8942)