What causes "multi-chunk stream with invalid chunk size of %d" in apache/beam?

Validate the writer side encodes chunk sizes as 0 (EOS) or a positive count

multi-chunk stream with invalid chunk size of %d

Error message

multi-chunk stream with invalid chunk size of %d

What it means

In a multi-chunked stream, chunk size markers must be >0 (elements) or 0 (end of stream). Any other negative value is invalid and produces this errors.Errorf. It signals wire-format corruption rather than an I/O failure.

Source

Thrown at sdks/go/pkg/beam/core/runtime/exec/datasource.go:381

					return nil, err
				}
				return &concatReStream{
					first: &FixedReStream{Buf: buf},
					next: &proxyReStream{
						open: func() (Stream, error) {
							r, err := n.state.OpenIterable(ctx, n.SID, token)
							if err != nil {
								return nil, err
							}
							// We can't re-use the original bcr, since we may get new iterables,
							// or multiple of them at the same time, but we can re-use the count itself.
							r = &byteCountReader{reader: r, count: bcr.count}
							return &elementStream{r: r, ec: cv}, nil
						},
					},
				}, nil
			default:
				return nil, errors.Errorf("multi-chunk stream with invalid chunk size of %d", chunk)
			}
		}
	default:
		return nil, errors.Errorf("received stream with marker size of %d", size)
	}
}

func readStreamToBuffer(cv ElementDecoder, r io.Reader, size int64, buf []FullValue) ([]FullValue, error) {
	for i := int64(0); i < size; i++ {
		value, err := cv.Decode(r)
		if err != nil {
			return nil, errors.Wrap(err, "stream value decode failed")
		}
		buf = append(buf, *value)
	}
	return buf, nil
}

View on GitHub (pinned to 12126d8942)

Solutions

Validate the writer side encodes chunk sizes as 0 (EOS) or a positive count
Check for coder/stream misalignment from an earlier failed decode in the same buffer
Confirm all pipeline stages use matching Beam SDK versions
Capture the raw bytes (debug logging) to diagnose the corrupted framing

Defensive patterns

Strategy: validation

Validate before calling

// writer side: only emit chunk sizes >0 or 0 for EOS
if chunkSize < 0 {
    return fmt.Errorf("refusing to write invalid chunk size %d", chunkSize)
}

Try / catch

if strings.Contains(err.Error(), "invalid chunk size") {
    return fmt.Errorf("wire-format corruption detected: %w", err)
}

Prevention

Never write negative chunk markers other than the documented values
Add round-trip tests for multi-chunk encoding of large iterables
Check for corruption sources in the transport layer

When it happens

Trigger: DecodeVarInt in the multi-chunk loop returns a chunk value < -1 (not 0 and not >0); e.g. a -2 or other negative marker due to corrupt or misaligned bytes.

Common situations: Malformed data from an incompatible writer implementation; bit-rot or corruption in shuffle transport; custom coders writing non-standard framing.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2b201fb151f254b3. Report an issue: GitHub.