apache/beam · error · IOException

invalid length " + length

Error message

invalid length " + length

What it means

ByteArrayCoder.decode throws this IOException when the VarInt length prefix read from the input stream is negative. A negative length means the encoded byte array is corrupt, truncated, or was written by an incompatible coder, so decoding cannot proceed.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ByteArrayCoder.java:105

      } else {
        outStream.write(value);
      }
    }
  }

  @Override
  public byte[] decode(InputStream inStream) throws IOException, CoderException {
    return decode(inStream, Context.NESTED);
  }

  @Override
  public byte[] decode(InputStream inStream, Context context) throws IOException, CoderException {
    if (context.isWholeStream) {
      return StreamUtils.getBytesWithoutClosing(inStream);
    } else {
      int length = VarInt.decodeInt(inStream);
      if (length < 0) {
        throw new IOException("invalid length " + length);
      }
      byte[] value = new byte[length];
      ByteStreams.readFully(inStream, value);
      return value;
    }
  }

  @Override
  public void verifyDeterministic() {}

  /**
   * {@inheritDoc}
   *
   * @return objects that are equal if the two arrays contain the same bytes.
   */
  @Override
  public Object structuralValue(byte[] value) {
    return new StructuralByteArray(value);

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify the input stream position is at the start of a ByteArrayCoder-encoded element, not mid-record
2. Ensure the same coder and Context were used on the writer side
3. Recover by re-reading from a checkpoint or re-generating the corrupted input data
4. Wrap decode in try-catch for IOException/CoderException and treat the record as corrupt rather than crashing the pipeline

Example fix

// before
byte[] value = coder.decode(inStream, Context.NOT_WHOLE_STREAM);
// after
try {
  byte[] value = coder.decode(inStream, Context.NOT_WHOLE_STREAM);
} catch (IOException e) {
  LOG.warn("Skipping corrupt byte[] record", e);
  return null; // skip / dead-letter the record
}

// Cannot validate without consuming the stream; log position before decoding.
long pos = (inStream instanceof FileInputStream) ? ((FileInputStream) inStream).getChannel().position() : -1;

try { byte[] v = ByteArrayCoder.of().decode(in, Context.NOT_WHOLE_STREAM); } catch (IOException | CoderException e) { /* corrupt input: skip or dead-letter */ }

When it happens

Trigger: Calling ByteArrayCoder.decode(inStream, context) with a non-whole-stream Context on a stream whose VarInt length prefix decodes to a negative value (truncated or misaligned data).

Common situations: Corrupt or truncated pipeline data, reading from the wrong stream offset, decoding a stream written with a different coder or Beam version, sharded/concatenated encoded blobs read at the wrong boundary.

Related errors

• EOF encountered decoding 1 byte from input stream
• CoderException(exn)
• Invalid encoded string length: {}
• Expected a %s with components but received %s
• Error decoding payload

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d65ca2fd4a37abad. Report an issue: GitHub.