apache/beam · error

invalid varint in ordered list entry

Error message

invalid varint in ordered list entry

What it means

After reading up to 10 varint bytes for an ordered list entry's sort key, decodeOrderedListEntry validates the buffer with protowire.ConsumeVarint. If the consumed length is negative, the bytes don't form a valid protobuf varint (e.g. truncated 10-byte continuation). The entry is rejected rather than mis-decoded.

Solutions

1. Verify the state was written by a compatible Beam version using the same varint/coder conventions.
2. Clear/rebuild the affected state data if it came from an older or corrupted job.
3. Check the producer of the sort key — oversized or non-canonical varints (>10 bytes) are invalid in protobuf.
4. Add validation at write time so sort keys are encoded via protowire.AppendVarint.

Example fix

// before: raw manual encoding of sort key
buf = append(buf, byte(sortKey)) // breaks for large keys
// after
buf = protowire.AppendVarint(buf, uint64(sortKey)) // canonical varint

if len(buf) == 0 || len(buf) > 10 { return errors.New("sort key varint out of protobuf bounds") }
if n := protowire.ConsumeVarint(buf); n < 0 { return errors.New("invalid varint bytes") }

func isValidVarint(buf []byte) bool { _, consumed := protowire.ConsumeVarint(buf); return consumed >= 0 }

entry, err := ReadOrderedListState(ctx, ...)
if err != nil && strings.Contains(err.Error(), "invalid varint") {
    return nil, fmt.Errorf("state data corrupt or written by incompatible SDK: %w", err)
}

When it happens

Trigger: Reading an ordered list state entry whose sort-key bytes are malformed — buffer overflowed the 10-byte loop without a terminating byte (0x80 bit clear), or the bytes are corrupt.

Common situations: State data written by a different SDK version or coder layout; corrupted/truncated state payloads; a varint exceeding the maximum 10-byte protobuf encoding; bit-level bugs in custom state serialization.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• unexpected error reading varint
• cannot make a state provider for an unkeyed input
• invalid varint
• Aliased enumerations not currently supported.
• Any not yet supported

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4697c8a5e8e17809. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/exec/userstate.go:607

	// Read varint byte-by-byte.
	var buf [10]byte // max varint size
	var n int
	for n = 0; n < len(buf); n++ {
		_, err := r.Read(buf[n : n+1])
		if err != nil {
			if n == 0 {
				return state.OrderedListEntry{}, err
			}
			return state.OrderedListEntry{}, fmt.Errorf("unexpected error reading varint: %w", err)
		}
		if buf[n]&0x80 == 0 {
			n++
			break
		}
	}
	sortKey, consumed := protowire.ConsumeVarint(buf[:n])
	if consumed < 0 {
		return state.OrderedListEntry{}, fmt.Errorf("invalid varint in ordered list entry")
	}

	dec := MakeElementDecoder(coder.SkipW(c))
	fv, err := dec.Decode(r)
	if err != nil {
		return state.OrderedListEntry{}, err
	}
	return state.OrderedListEntry{SortKey: int64(sortKey), Value: fv.Elm}, nil
}

func (s *stateProvider) encodeKey(userStateID string, key any) ([]byte, error) {
	fv := FullValue{Elm: key}
	enc := MakeElementEncoder(coder.SkipW(s.keyCodersByID[userStateID]))
	var b bytes.Buffer
	err := enc.Encode(&fv, &b)
	if err != nil {
		return nil, err
	}

View on GitHub (pinned to 12126d8942)