apache/beam · error · IOException

invalid length " + length

Error message

invalid length " + length

What it means

ByteArrayCoder.decode throws this IOException when the VarInt length prefix read from the input stream is negative. A negative length means the encoded byte array is corrupt, truncated, or was written by an incompatible coder, so decoding cannot proceed.

Solutions

1. Verify the input stream position is at the start of a ByteArrayCoder-encoded element, not mid-record
2. Ensure the same coder and Context were used on the writer side
3. Recover by re-reading from a checkpoint or re-generating the corrupted input data
4. Wrap decode in try-catch for IOException/CoderException and treat the record as corrupt rather than crashing the pipeline

Example fix

// before
byte[] value = coder.decode(inStream, Context.NOT_WHOLE_STREAM);
// after
try {
  byte[] value = coder.decode(inStream, Context.NOT_WHOLE_STREAM);
} catch (IOException e) {
  LOG.warn("Skipping corrupt byte[] record", e);
  return null; // skip / dead-letter the record
}

// Cannot validate without consuming the stream; log position before decoding.
long pos = (inStream instanceof FileInputStream) ? ((FileInputStream) inStream).getChannel().position() : -1;

try { byte[] v = ByteArrayCoder.of().decode(in, Context.NOT_WHOLE_STREAM); } catch (IOException | CoderException e) { /* corrupt input: skip or dead-letter */ }

When it happens

Trigger: Calling ByteArrayCoder.decode(inStream, context) with a non-whole-stream Context on a stream whose VarInt length prefix decodes to a negative value (truncated or misaligned data).

Common situations: Corrupt or truncated pipeline data, reading from the wrong stream offset, decoding a stream written with a different coder or Beam version, sharded/concatenated encoded blobs read at the wrong boundary.

Related errors

• CoderException(exn)
• EOF encountered decoding 1 byte from input stream
• Error decoding payload
• Expected a with components but received
• Invalid encoded byte array

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d65ca2fd4a37abad. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ByteArrayCoder.java:105

      } else {
        outStream.write(value);
      }
    }
  }

  @Override
  public byte[] decode(InputStream inStream) throws IOException, CoderException {
    return decode(inStream, Context.NESTED);
  }

  @Override
  public byte[] decode(InputStream inStream, Context context) throws IOException, CoderException {
    if (context.isWholeStream) {
      return StreamUtils.getBytesWithoutClosing(inStream);
    } else {
      int length = VarInt.decodeInt(inStream);
      if (length < 0) {
        throw new IOException("invalid length " + length);
      }
      byte[] value = new byte[length];
      ByteStreams.readFully(inStream, value);
      return value;
    }
  }

  @Override
  public void verifyDeterministic() {}

  /**
   * {@inheritDoc}
   *
   * @return objects that are equal if the two arrays contain the same bytes.
   */
  @Override
  public Object structuralValue(byte[] value) {
    return new StructuralByteArray(value);

View on GitHub (pinned to 12126d8942)