apache/beam · error

unexpected error reading varint

Error message

unexpected error reading varint: %w

What it means

decodeOrderedListEntry reads a varint sort key byte-by-byte from the state API stream. If a read fails midway (after some bytes were consumed), the partial varint can't be returned cleanly, so the byte-level error is wrapped with this message. It signals an unexpected mid-entry failure of the ordered-list state reader.

Solutions

1. Retry the state read — the first byte's error is returned directly (likely EOF), so a wrapped error means a mid-entry interruption worth retrying.
2. Check the state API connection stability between SDK harness and runner (timeouts, keepalives, proxies).
3. Inspect the wrapped inner error (%w) to identify the transport failure.
4. Reduce list state entry sizes / verify the state server didn't truncate the response.

Example fix

// before: one-shot read without resilience
entry, err := ReadOrderedListState(ctx, ...)
// after: retry on transient mid-read failures
entry, err := ReadOrderedListState(ctx, ...)
if err != nil {
    return retryWithBackoff(ctx, func() error { _, e := ReadOrderedListState(ctx, ...); return e })
}

entry, err := ReadOrderedListState(ctx, ...)
if err != nil {
    var retriable = isTransportError(err) // inspect wrapped %w error
    if retriable { entry, err = retryWithBackoff(ctx, read) }
}

When it happens

Trigger: Reading an ordered list state entry where the underlying Read call fails after the first byte — network drop to the state/harness service, stream closed mid-entry, or truncated state response.

Common situations: Dataflow/runner harness connection resets during state reads; state service timeouts; reading past the end of a malformed stream; transient gRPC/HTTP failures on the state API channel.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• invalid varint in ordered list entry
• cannot make a state provider for an unkeyed input
• varint not terminated
• varint overflow
• varint too long

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/53d02d12882bf002. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/exec/userstate.go:598

	if err := enc.Encode(&fv, &buf); err != nil {
		return err
	}
	_, err := w.Write(buf.Bytes())
	return err
}

// decodeOrderedListEntry reads varint(sortKey) || coder_encoded(value) from r.
func decodeOrderedListEntry(r io.Reader, c *coder.Coder) (state.OrderedListEntry, error) {
	// Read varint byte-by-byte.
	var buf [10]byte // max varint size
	var n int
	for n = 0; n < len(buf); n++ {
		_, err := r.Read(buf[n : n+1])
		if err != nil {
			if n == 0 {
				return state.OrderedListEntry{}, err
			}
			return state.OrderedListEntry{}, fmt.Errorf("unexpected error reading varint: %w", err)
		}
		if buf[n]&0x80 == 0 {
			n++
			break
		}
	}
	sortKey, consumed := protowire.ConsumeVarint(buf[:n])
	if consumed < 0 {
		return state.OrderedListEntry{}, fmt.Errorf("invalid varint in ordered list entry")
	}

	dec := MakeElementDecoder(coder.SkipW(c))
	fv, err := dec.Decode(r)
	if err != nil {
		return state.OrderedListEntry{}, err
	}
	return state.OrderedListEntry{SortKey: int64(sortKey), Value: fv.Elm}, nil
}

View on GitHub (pinned to 12126d8942)