What causes "failed to decode union value '%v' for key %v" in apache/beam?

Inspect the wrapped inner error to find the actual decode failure.

failed to decode union value '%v' for key %v

Error message

failed to decode union value '%v' for key %v

What it means

During CoGBK union value decoding, the reader encounters a key matching this union's index but the encoded value bytes cannot be decoded by the union's decoder. The error wraps the underlying decode failure with the offending bytes and key for diagnosis.

Source

Thrown at sdks/go/pkg/beam/core/runtime/exec/cogbk.go:172

func (f *filterStream) Read() (*FullValue, error) {
	for {
		elm, err := f.real.Read()
		if err != nil {
			return nil, err
		}

		key := elm.Elm.(int)
		value := elm.Elm2.([]byte)

		// Transform KV<int,[]byte> into V iff key == N

		if key != f.n {
			continue // skip other keys
		}

		v, err := f.dec.Decode(bytes.NewReader(value))
		if err != nil {
			return nil, errors.Wrapf(err, "failed to decode union value '%v' for key %v", value, key)
		}
		v.Timestamp = elm.Timestamp
		v.Windows = elm.Windows
		return v, nil
	}
}

View on GitHub (pinned to 12126d8942)

Solutions

Inspect the wrapped inner error to find the actual decode failure.
Ensure producer and consumer use the same coder/schema version for the union value.
Re-run or regenerate the data if it was written by an incompatible pipeline version.

Defensive patterns

Strategy: try-catch

Try / catch

v, err := ds.Read(ctx)
if err != nil {
	var decErr interface{ Unwrap() error }
	if errors.As(err, &wrapped) && strings.Contains(err.Error(), "failed to decode union value") {
		// log key/bytes and fail fast; data is schema-incompatible
	}
	return err
}

Prevention

Keep coder/schema versions in sync between writers and readers.
Avoid changing union value coders without migrating existing data.
Log the inner decode error (errors.Unwrap) to identify the coder mismatch.

When it happens

Trigger: Read() on a CoGBK/union stream where f.dec.Decode fails on the value bytes — typically when the wire bytes were written with a different coder version or schema than the one being used to read.

Common situations: Schema evolution across pipeline versions (old data read by new coder), cross-language pipelines whose union coders disagree, corrupted or truncated blob in the CoGBK shard.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4b21e19b78820707. Report an issue: GitHub.