What causes "Invalid encoded byte array" in apache/beam?

Check hasRemainingEncodedBytes() before each read and stop when it returns false.

apache/beam · error · IllegalArgumentException

Invalid encoded byte array

Error message

Invalid encoded byte array

What it means

readBytes (and readBytesDecreasing) begins decoding the next segment from the internal encoded store. If the store is empty or the remaining bytes in the first array are exhausted (length - firstArrayPosition <= 0), the encoded stream is malformed or fully consumed, so it throws IllegalArgumentException("Invalid encoded byte array").

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/OrderedCode.java:383

  }

  /**
   * Returns the next byte array item from its encoded byte array store and removes the item from
   * the store.
   *
   * @see #writeBytes(byte[])
   */
  public byte[] readBytes() {
    return readBytes(false);
  }

  public byte[] readBytesDecreasing() {
    return readBytes(true);
  }

  private byte[] readBytes(boolean invert) {
    if (encodedArrays.isEmpty() || (encodedArrays.get(0).length - firstArrayPosition <= 0)) {
      throw new IllegalArgumentException("Invalid encoded byte array");
    }

    // Determine the length of the decoded array
    // We only scan up to "length-2" since a valid string must end with
    // a two character terminator: 'ESCAPE1 SEPARATOR'
    byte[] store = encodedArrays.get(0);
    int decodedLength = 0;
    boolean valid = false;
    int i = firstArrayPosition;
    while (i < store.length - 1) {
      byte b = store[i++];
      if (b == convert(invert, ESCAPE1)) {
        b = store[i++];
        if (b == convert(invert, SEPARATOR)) {
          valid = true;
          break;
        } else if (b == convert(invert, NULL_CHARACTER)) {
          decodedLength++;

View on GitHub (pinned to 12126d8942)

Solutions

Check hasRemainingEncodedBytes() before each read and stop when it returns false.
Ensure the byte array being decoded was produced entirely by the matching OrderedCode write methods without external modification.
Pair every read* call one-to-one with its write* call in the same order used at encode time.

Example fix

// before
byte[] b = oc.readBytes();
// after
if (oc.hasRemainingEncodedBytes()) {
  byte[] b = oc.readBytes();
}

Defensive patterns

Strategy: try-catch

Validate before calling

if (!oc.hasRemainingEncodedBytes()) { return null; /* or throw domain error */ }

Try / catch

try { return oc.readBytes(); } catch (IllegalArgumentException e) { if (e.getMessage().equals("Invalid encoded byte array")) { throw new CorruptKeyException(e); } throw e; }

Prevention

Check hasRemainingEncodedBytes() before each read
Never slice or re-encode OrderedCode output manually
Persist encoded keys as binary-safe values

When it happens

Trigger: Calling readBytes/readBytesDecreasing more times than writeBytes was called, or decoding a buffer that was truncated/corrupted in transit, so nothing remains to read.

Common situations: Deserializing keys stored in an external system (Spanner keys, ordering keys) that were hand-trimmed, round-tripped through a different encoder, or sliced at wrong boundaries.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8c6509ea9d720ec1. Report an issue: GitHub.