apache/beam · error
invalid varintz encoding for: %v
Error message
invalid varintz encoding for: %v
What it means
decVarIntZ decodes bytes with binary.Varint (zig-zag). If Varint returns size <= 0 the data is empty, truncated, or malformed, and this error is returned. It indicates the byte stream doesn't conform to the varintz scheme, not that the decoded value is out of range.
Source
Thrown at sdks/go/pkg/beam/core/runtime/coderx/varint.go:74
val = int64(n)
case int16:
val = int64(n)
case int32:
val = int64(n)
case int64:
val = n
default:
panic(fmt.Sprintf("received unknown value type: want a signed integer:, got %T", n))
}
ret := make([]byte, binary.MaxVarintLen64)
size := binary.PutVarint(ret, val)
return ret[:size]
}
func decVarIntZ(t reflect.Type, data []byte) (typex.T, error) {
n, size := binary.Varint(data)
if size <= 0 {
return nil, errors.Errorf("invalid varintz encoding for: %v", data)
}
switch t.Kind() {
case reflect.Int:
return int(n), nil
case reflect.Int8:
return int8(n), nil
case reflect.Int16:
return int16(n), nil
case reflect.Int32:
return int32(n), nil
case reflect.Int64:
return n, nil
default:
panic(fmt.Sprintf("unreachable statement: expected a signed integer, got %v", t))
}
}
func encVarUintZ(v typex.T) []byte {View on GitHub (pinned to 12126d8942)
Solutions
- Ensure decoding uses the same coder (NewVarIntZ) used to encode the data.
- Check buffer boundaries — a truncated record yields size <= 0; fix the framing/length prefix.
- Compare encodings: varintz (binary.Varint) differs from Beam's standard varint; don't mix schemes.
- Hex-dump the offending bytes to verify the first byte is a valid varint continuation sequence.
Defensive patterns
Strategy: validation
Validate before calling
// validate a buffer before varintz decode
func validVarIntZ(data []byte) bool {
_, size := binary.Varint(data)
return size > 0
} Type guard
func isVarIntZCoder(c *coder.CustomCoder) bool { return c.Name == "varintz" } Try / catch
val, err := coder.Decode(data)
if err != nil {
if strings.Contains(err.Error(), "invalid varintz encoding") {
log.Warnf("bad varintz record: % x", data)
return nil, nil // dead-letter instead
}
return nil, err
} Prevention
- Encode and decode with the same NewVarIntZ coder; never mix with varuintz.
- Frame records with explicit lengths so decoders never see truncated buffers.
- Note varintz uses Go's zig-zag binary.Varint, not Beam's standard varint — don't cross decode.
- Checksum stored records to catch corruption before decode.
When it happens
Trigger: Decoding (via the 'varintz' custom coder) byte slices not produced by encVarIntZ — wrong coder, truncated buffer, corrupted storage.
Common situations: Mixing varintz and varuintz encoded data, reading partial records from a stream, cross-version data with a changed encoding.
Understand the failure class
Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.
Related errors
- invalid varuintz encoding for: %v
- invalid float encoding for: %v
- VarLong not terminated.
- ReadSimpleRowHeader reading nils count: %v, %v
- error decoding bool: %v
AI-assisted analysis of apache/beam@12126d8942 (2026-09-13).
Data as JSON: /api/errors/5e6bf215f5150328.
Report an issue: GitHub.