{"record":{"id":"681f9a5b9931d329","repo":"apache/beam","slug":"stream-chunk-size-decoding-failed","errorCode":null,"errorMessage":"stream chunk size decoding failed","messagePattern":"stream chunk size decoding failed","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/core/runtime/exec/datasource.go","lineNumber":343,"sourceCode":"\t\t}\n\t}\n\n\tswitch {\n\tcase size >= 0:\n\t\t// Single chunk streams are fully read in and buffered in memory.\n\t\tbuf := make([]FullValue, 0, size)\n\t\tbuf, err = readStreamToBuffer(cv, bcr, int64(size), buf)\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t\treturn &FixedReStream{Buf: buf}, nil\n\tcase size == -1:\n\t\t// Multi-chunked stream.\n\t\tvar buf []FullValue\n\t\tfor {\n\t\t\tchunk, err := coder.DecodeVarInt(bcr.reader)\n\t\t\tif err != nil {\n\t\t\t\treturn nil, errors.Wrap(err, \"stream chunk size decoding failed\")\n\t\t\t}\n\t\t\t// All done, escape out.\n\t\t\tswitch {\n\t\t\tcase chunk == 0: // End of stream, return buffer.\n\t\t\t\treturn &FixedReStream{Buf: buf}, nil\n\t\t\tcase chunk > 0: // Non-zero chunk, read that many elements from the stream, and buffer them.\n\t\t\t\tchunkBuf := make([]FullValue, 0, chunk)\n\t\t\t\tchunkBuf, err = readStreamToBuffer(cv, bcr, chunk, chunkBuf)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn nil, err\n\t\t\t\t}\n\t\t\t\tbuf = append(buf, chunkBuf...)\n\t\t\tcase chunk == -1: // State backed iterable!\n\t\t\t\tchunk, err := coder.DecodeVarInt(bcr.reader)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn nil, err\n\t\t\t\t}\n\t\t\t\ttoken, err := ioutilx.ReadN(bcr.reader, (int)(chunk))","sourceCodeStart":325,"sourceCodeEnd":361,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/core/runtime/exec/datasource.go#L325-L361","documentation":"For a multi-chunked stream (size marker == -1), each chunk is prefixed with a varint chunk size. If coder.DecodeVarInt fails while reading that prefix, the error is wrapped as 'stream chunk size decoding failed'. This indicates truncation or corruption within a multi-chunk iterable.","triggerScenarios":"Reading a large iterable encoded as multiple chunks; DecodeVarInt on bcr.reader returns EOF or invalid encoding while scanning chunk headers.","commonSituations":"Very large GBK results split into chunks whose data got truncated; runner-side buffer boundaries cutting a stream; misaligned coders from a prior failed decode.","solutions":["Verify the complete stream was transferred (check runner/shuffle logs for truncation)","Ensure consistent coder framing between writer and reader stages","Retry the bundle in case of a transient transport failure","Reduce element/iterable size if buffers are being dropped at transport limits"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if strings.Contains(err.Error(), \"stream chunk size decoding failed\") {\n    // treat as data corruption; fail fast or re-fetch bundle\n    return fmt.Errorf(\"corrupt multi-chunk stream: %w\", err)\n}","preventionTips":["Keep element streams intact across transport boundaries","Confirm writer emits 0-terminated multi-chunk framing","Match SDK versions across all workers"],"tags":["beam","go","decoding","stream"],"backgroundTag":"unexpected-response-shape","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}