apache/beam · error

stream chunk size decoding failed

Error message

stream chunk size decoding failed

What it means

For a multi-chunked stream (size marker == -1), each chunk is prefixed with a varint chunk size. If coder.DecodeVarInt fails while reading that prefix, the error is wrapped as 'stream chunk size decoding failed'. This indicates truncation or corruption within a multi-chunk iterable.

Solutions

1. Verify the complete stream was transferred (check runner/shuffle logs for truncation)
2. Ensure consistent coder framing between writer and reader stages
3. Retry the bundle in case of a transient transport failure
4. Reduce element/iterable size if buffers are being dropped at transport limits

if strings.Contains(err.Error(), "stream chunk size decoding failed") {
    // treat as data corruption; fail fast or re-fetch bundle
    return fmt.Errorf("corrupt multi-chunk stream: %w", err)
}

When it happens

Trigger: Reading a large iterable encoded as multiple chunks; DecodeVarInt on bcr.reader returns EOF or invalid encoding while scanning chunk headers.

Common situations: Very large GBK results split into chunks whose data got truncated; runner-side buffer boundaries cutting a stream; misaligned coders from a prior failed decode.

Related errors

• stream size decoding failed
• decodeMultiChunkStream chunk size decoding failed
• decodeStream value decode failed on close
• decoding a *
• decoding a

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/681f9a5b9931d329. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/exec/datasource.go:343

		}
	}

	switch {
	case size >= 0:
		// Single chunk streams are fully read in and buffered in memory.
		buf := make([]FullValue, 0, size)
		buf, err = readStreamToBuffer(cv, bcr, int64(size), buf)
		if err != nil {
			return nil, err
		}
		return &FixedReStream{Buf: buf}, nil
	case size == -1:
		// Multi-chunked stream.
		var buf []FullValue
		for {
			chunk, err := coder.DecodeVarInt(bcr.reader)
			if err != nil {
				return nil, errors.Wrap(err, "stream chunk size decoding failed")
			}
			// All done, escape out.
			switch {
			case chunk == 0: // End of stream, return buffer.
				return &FixedReStream{Buf: buf}, nil
			case chunk > 0: // Non-zero chunk, read that many elements from the stream, and buffer them.
				chunkBuf := make([]FullValue, 0, chunk)
				chunkBuf, err = readStreamToBuffer(cv, bcr, chunk, chunkBuf)
				if err != nil {
					return nil, err
				}
				buf = append(buf, chunkBuf...)
			case chunk == -1: // State backed iterable!
				chunk, err := coder.DecodeVarInt(bcr.reader)
				if err != nil {
					return nil, err
				}
				token, err := ioutilx.ReadN(bcr.reader, (int)(chunk))

View on GitHub (pinned to 12126d8942)