apache/beam · error · CoderException

unexpected extra bytes after decoding

Error message

 unexpected extra bytes after decoding 

What it means

decodeFromByteArray checks that the Coder consumed the entire input; if bytes remain after decoding (stream.available() != 0) it throws CoderException naming the leftover count and the decoded value. It means the encoded byte array does not exactly match what the coder would produce — a framing or coder-mismatch problem.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/CoderUtils.java:104

      coder.encode(value, new UnownedOutputStream(stream), context);
    } catch (IOException exn) {
      Throwables.propagateIfPossible(exn, CoderException.class);
      throw new IllegalArgumentException("Forbidden IOException when writing to OutputStream", exn);
    }
  }

  /** Decodes the given bytes using the specified Coder, and returns the resulting decoded value. */
  public static <T> T decodeFromByteArray(Coder<T> coder, byte[] encodedValue)
      throws CoderException {
    return decodeFromByteArray(coder, encodedValue, Coder.Context.OUTER);
  }

  public static <T> T decodeFromByteArray(
      Coder<T> coder, byte[] encodedValue, Coder.Context context) throws CoderException {
    try (ExposedByteArrayInputStream stream = new ExposedByteArrayInputStream(encodedValue)) {
      T result = decodeFromSafeStream(coder, stream, context);
      if (stream.available() != 0) {
        throw new CoderException(
            stream.available() + " unexpected extra bytes after decoding " + result);
      }
      return result;
    }
  }

  /**
   * Decodes a value from the given ByteString, validating that no bytes are remaining once decoded.
   */
  public static <T> T decodeFromByteString(Coder<T> coder, ByteString encodedValue)
      throws IOException {
    return decodeFromByteString(coder, encodedValue, Coder.Context.OUTER);
  }

  /**
   * Decodes a value from the given ByteString using a given context, validating that no bytes are
   * remaining once decoded.
   */

View on GitHub (pinned to 12126d8942)

Solutions

1. Decode with the exact same Coder instance/registration used to encode
2. Ensure the byte array contains exactly one encoded element — strip trailing newlines/terminators or slice the exact length
3. If values were concatenated, decode iteratively with the coder consuming one element at a time instead of one decode of all bytes
4. Verify coder registries/coder clouds match between writer and reader stages

Example fix

// before
byte[] all = concat(e1, e2);
MyValue v = CoderUtils.decodeFromByteArray(coder, all); // extra bytes
// after
MyValue v1 = CoderUtils.decodeFromByteArray(coder, e1);
MyValue v2 = CoderUtils.decodeFromByteArray(coder, e2);

// before decoding
if (bytes.length == 0 || bytes.length < minEncodedSize) {
  throw new IllegalArgumentException("buffer too small/empty for one encoded element");
}
// after decoding in dev tests:
// assert re-encode(decoded).length == bytes.length

try {
  T v = CoderUtils.decodeFromByteArray(coder, bytes);
} catch (CoderException e) {
  // coder mismatch or trailing bytes; re-slice or use correct coder
}

When it happens

Trigger: Calling CoderUtils.decodeFromByteArray(coder, bytes) where bytes were produced by a different coder, with trailing padding/terminators (e.g. extra newline from a text-based format), or from concatenating multiple encoded values into one call.

Common situations: Encoding with one coder version and decoding after a coder change; appending separators or length prefixes manually; decoding an element from a batch without slicing its exact byte range.

Related errors

• %s does not support non zero terminator values. Received str
• Cannot provide SerializableCoder because {} does not impleme
• Java Serialization may be non-deterministic.
• Forbidden IOException when writing to OutputStream
• Forbidden IOException when reading from InputStream

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ed74176dfec242e3. Report an issue: GitHub.