{"record":{"id":"c30c1f5d4357abd7","repo":"apache/beam","slug":"varint-too-long-c30c1f","errorCode":null,"errorMessage":"varint too long","messagePattern":"varint too long","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/core/graph/coder/varint.go","lineNumber":28,"sourceCode":"// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\npackage coder\n\nimport (\n\t\"io\"\n\n\t\"github.com/apache/beam/sdks/v2/go/pkg/beam/core/util/ioutilx\"\n\t\"github.com/apache/beam/sdks/v2/go/pkg/beam/internal/errors\"\n)\n\n// ErrVarIntTooLong indicates a data corruption issue that needs special\n// handling by callers of decode. TODO(herohde): have callers perform\n// this special handling.\nvar ErrVarIntTooLong = errors.New(\"varint too long\")\n\n// EncodeVarUint64 encodes an uint64.\nfunc EncodeVarUint64(value uint64, w io.Writer) error {\n\tret := make([]byte, 0, 8)\n\tfor {\n\t\t// Encode next 7 bits + terminator bit\n\t\tbits := value & 0x7f\n\t\tvalue >>= 7\n\n\t\tvar mask uint64\n\t\tif value != 0 {\n\t\t\tmask = 0x80\n\t\t}\n\t\tret = append(ret, (byte)(bits|mask))\n\t\tif value == 0 {\n\t\t\t_, err := ioutilx.WriteUnsafe(w, ret)\n\t\t\treturn err\n\t\t}","sourceCodeStart":10,"sourceCodeEnd":46,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/core/graph/coder/varint.go#L10-L46","documentation":"ErrVarIntTooLong is a sentinel error from DecodeVarUint64 in the Beam Go coder package. While decoding a varint (as used in the Beam Fn API runner wire protocol), either the shift exceeded 64 bits or the 64th-bit pattern was invalid, meaning the encoded bytes do not represent a valid uint64 varint — the code path treats it as data corruption.","triggerScenarios":"Decoding a byte stream where a varint has more than 10 continuation bytes, or the final byte sets bit patterns pushing shift past 63 (shift >= 64, or shift == 63 with bits > 1), typically from misaligned or truncated data streams.","commonSituations":"Corrupted pipeline data between runner harnesses, deserializing coders from the wrong offset after a coder mismatch, hand-rolling varint encoding incompatible with Beam's LEB128-style format, or version-skewed runner/SDK pairs.","solutions":["Check that reader/writer coder versions match between the harness and SDK (no version skew).","Verify stream alignment: re-check where decoding starts; a desync makes the next varint read garbage.","Ensure custom encoders use Beam's varint format (7 bits per byte, high bit = continuation) matching EncodeVarUint64.","If corruption is suspected, add checksums/re-encode at the source and surface the surrounding byte context for debugging."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Validate varint byte length before decoding custom data\nif len(buf) == 0 || countContinuationBytes(buf) > 10 {\n    return errors.New(\"invalid varint encoding\")\n}","typeGuard":null,"tryCatchPattern":"n, err := coder.DecodeVarUint64(r)\nif errors.Is(err, coder.ErrVarIntTooLong) {\n    // treat as corrupted data: resync stream, log offset, fail or skip record\n}","preventionTips":["Keep runner and SDK versions aligned to avoid coder mismatches.","Use EncodeVarUint64 for all custom varint writes.","Detect decode desync early by validating record framing before decoding fields."],"tags":["go","beam","coder","data-corruption","serialization"],"backgroundTag":"data-corruption","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}