apache/beam · error

invalid varint

Error message

invalid varint

What it means

Prism's local decoder wraps protowire.ConsumeVarint; if the wire library reports a negative byte count, the bytes at the cursor are not a valid protobuf varint (malformed or truncated). The decoder panics because timer payload decoding cannot safely continue from a broken wire format.

Solutions

1. Ensure window coders used with timers are length-prefixed (prism requires this; see the singleWindowExtractor setup)
2. Match SDK harness and prism versions to avoid timer wire-format skew
3. Dump the raw timer bytes at failure to check whether the payload is truncated or misaligned
4. If a custom coder precedes the varint, verify it consumes exactly the right number of bytes so the cursor stays aligned

Example fix

// before: non-length-prefixed custom window coder corrupts alignment
singleWindowExtractor = func(d *decoder) typex.Window { return d.CustomWindow() }
// after: force length prefix so the varint cursor stays aligned
singleWindowExtractor = func(d *decoder) typex.Window { return d.CustomWindowLengthPrefixed() }

// Sanity check remaining bytes before decoding
if len(raw) == 0 || d.cursor >= len(d.raw) {
    return io.ErrUnexpectedEOF
}

When it happens

Trigger: decoder.Varint called (directly or via IntervalWindow, CustomWindowLengthPrefixed, Bytes, Pane) on raw timer bytes whose next field is not a well-formed varint — truncated payloads, misaligned cursor, or a length-prefix mismatch earlier in the stream.

Common situations: Timer payload corruption, coder mismatches between SDK and runner (e.g. non-length-prefixed custom window coders), or SDK/runner version skew changing the timer wire format.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• error decoding watermarks
• bad type: , want
• base64 decode for failed
• computeFacts: unable to check
• couldn't decode characteristic for variant

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2f54fec8abc99510. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/prism/internal/engine/timers.go:144

			if !yield(timerRet{keyBytes, tag, elms, ws}) {
				return // Halt iteration if yield returns false.
			}
			// Otherwise continue handling the remaining bytes.
			raw = d.UnusedBytes()
		}
	}
}

type decoder struct {
	raw    []byte
	cursor int
}

// Varint consumes a varint from the bytes, returning the decoded length.
func (d *decoder) Varint() (l int64) {
	v, n := protowire.ConsumeVarint(d.raw[d.cursor:])
	if n < 0 {
		panic("invalid varint")
	}
	d.cursor += n
	return int64(v)
}

// Uint64 decodes a value of type uint64.
func (d *decoder) Uint64() uint64 {
	defer func() {
		d.cursor += 8
	}()
	return binary.BigEndian.Uint64(d.raw[d.cursor : d.cursor+8])
}

func (d *decoder) Timestamp() mtime.Time {
	msec := d.Uint64()
	return mtime.Time((int64)(msec) + math.MinInt64)
}

View on GitHub (pinned to 12126d8942)