apache/beam · error

received stream with marker size of %d

Error message

received stream with marker size of %d

What it means

The first int32 in a nested stream must be a known marker: -1 (multi-chunked), 0, or a positive element count. Receiving any other value produces this errors.Errorf, meaning the stream framing is unrecognized — typically a coder/protocol mismatch.

Source

Thrown at sdks/go/pkg/beam/core/runtime/exec/datasource.go:385

					next: &proxyReStream{
						open: func() (Stream, error) {
							r, err := n.state.OpenIterable(ctx, n.SID, token)
							if err != nil {
								return nil, err
							}
							// We can't re-use the original bcr, since we may get new iterables,
							// or multiple of them at the same time, but we can re-use the count itself.
							r = &byteCountReader{reader: r, count: bcr.count}
							return &elementStream{r: r, ec: cv}, nil
						},
					},
				}, nil
			default:
				return nil, errors.Errorf("multi-chunk stream with invalid chunk size of %d", chunk)
			}
		}
	default:
		return nil, errors.Errorf("received stream with marker size of %d", size)
	}
}

func readStreamToBuffer(cv ElementDecoder, r io.Reader, size int64, buf []FullValue) ([]FullValue, error) {
	for i := int64(0); i < size; i++ {
		value, err := cv.Decode(r)
		if err != nil {
			return nil, errors.Wrap(err, "stream value decode failed")
		}
		buf = append(buf, *value)
	}
	return buf, nil
}

// FinishBundle resets the source.
func (n *DataSource) FinishBundle(ctx context.Context) error {
	n.mu.Lock()
	defer n.mu.Unlock()

View on GitHub (pinned to 12126d8942)

Solutions

1. Align SDK versions between pipeline submission and workers
2. Audit any custom coders for extra/missing bytes that offset the stream
3. Log the offending size value and compare against the expected framing (-1, 0, or N)
4. Check runner-side data serialization for recent changes

// validate stream marker before handing off to Beam internals
// marker must be -1, 0, or a positive count
if !(size == -1 || size >= 0) { /* impossible for int32, but for custom formats: */ }
// for custom encoders: assert recognized marker
if size != -1 && size < 0 {
    return fmt.Errorf("unsupported stream marker %d", size)
}

if strings.Contains(err.Error(), "marker size of") {
    return fmt.Errorf("unrecognized stream framing: %w", err)
}

When it happens

Trigger: makeReStream's size switch falls to `default` because size is neither -1, 0, nor >0 (e.g. -2 or another negative value).

Common situations: Runner and SDK disagree on stream encoding (version skew); custom element coders emitting stray bytes that shift the size marker; data corruption in transit.

Related errors

• multi-chunk stream with invalid chunk size of %d
• unexpected type kind %v
• Unknown ValueKind number: {}
• Unknown type
• Type interface{} isn't a supported PCollection type

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/22f76310e3477774. Report an issue: GitHub.