{"record":{"id":"2b201fb151f254b3","repo":"apache/beam","slug":"multi-chunk-stream-with-invalid-chunk-size-of-d","errorCode":null,"errorMessage":"multi-chunk stream with invalid chunk size of %d","messagePattern":"multi-chunk stream with invalid chunk size of (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/core/runtime/exec/datasource.go","lineNumber":381,"sourceCode":"\t\t\t\t\treturn nil, err\n\t\t\t\t}\n\t\t\t\treturn &concatReStream{\n\t\t\t\t\tfirst: &FixedReStream{Buf: buf},\n\t\t\t\t\tnext: &proxyReStream{\n\t\t\t\t\t\topen: func() (Stream, error) {\n\t\t\t\t\t\t\tr, err := n.state.OpenIterable(ctx, n.SID, token)\n\t\t\t\t\t\t\tif err != nil {\n\t\t\t\t\t\t\t\treturn nil, err\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t// We can't re-use the original bcr, since we may get new iterables,\n\t\t\t\t\t\t\t// or multiple of them at the same time, but we can re-use the count itself.\n\t\t\t\t\t\t\tr = &byteCountReader{reader: r, count: bcr.count}\n\t\t\t\t\t\t\treturn &elementStream{r: r, ec: cv}, nil\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t}, nil\n\t\t\tdefault:\n\t\t\t\treturn nil, errors.Errorf(\"multi-chunk stream with invalid chunk size of %d\", chunk)\n\t\t\t}\n\t\t}\n\tdefault:\n\t\treturn nil, errors.Errorf(\"received stream with marker size of %d\", size)\n\t}\n}\n\nfunc readStreamToBuffer(cv ElementDecoder, r io.Reader, size int64, buf []FullValue) ([]FullValue, error) {\n\tfor i := int64(0); i < size; i++ {\n\t\tvalue, err := cv.Decode(r)\n\t\tif err != nil {\n\t\t\treturn nil, errors.Wrap(err, \"stream value decode failed\")\n\t\t}\n\t\tbuf = append(buf, *value)\n\t}\n\treturn buf, nil\n}\n","sourceCodeStart":363,"sourceCodeEnd":399,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/core/runtime/exec/datasource.go#L363-L399","documentation":"In a multi-chunked stream, chunk size markers must be >0 (elements) or 0 (end of stream). Any other negative value is invalid and produces this errors.Errorf. It signals wire-format corruption rather than an I/O failure.","triggerScenarios":"DecodeVarInt in the multi-chunk loop returns a chunk value < -1 (not 0 and not >0); e.g. a -2 or other negative marker due to corrupt or misaligned bytes.","commonSituations":"Malformed data from an incompatible writer implementation; bit-rot or corruption in shuffle transport; custom coders writing non-standard framing.","solutions":["Validate the writer side encodes chunk sizes as 0 (EOS) or a positive count","Check for coder/stream misalignment from an earlier failed decode in the same buffer","Confirm all pipeline stages use matching Beam SDK versions","Capture the raw bytes (debug logging) to diagnose the corrupted framing"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// writer side: only emit chunk sizes >0 or 0 for EOS\nif chunkSize < 0 {\n    return fmt.Errorf(\"refusing to write invalid chunk size %d\", chunkSize)\n}","typeGuard":null,"tryCatchPattern":"if strings.Contains(err.Error(), \"invalid chunk size\") {\n    return fmt.Errorf(\"wire-format corruption detected: %w\", err)\n}","preventionTips":["Never write negative chunk markers other than the documented values","Add round-trip tests for multi-chunk encoding of large iterables","Check for corruption sources in the transport layer"],"tags":["beam","go","wire-format","validation"],"backgroundTag":"unexpected-response-shape","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}