What causes "failed encoding key for %v: %v" in apache/beam?

Inspect the wrapped inner error to identify the failing coder and fix the coder implementation.

failed encoding key for

Error message

failed encoding key for %v: %v

What it means

During CoGBK reprocessing by window in the direct runner, keys are re-encoded into a new group per merged window. If the key coder fails to encode the key element, reprocessByWindow wraps the underlying coder error as "failed encoding key for %v: %v". This indicates the key's encoded bytes cannot be produced, typically due to an unsupported or misregistered key coder.

Solutions

Inspect the wrapped inner error to identify the failing coder and fix the coder implementation.
Ensure the key type has a valid registered coder or use a natively encodable key type (string, int, etc.).
Check the custom coder's Encode for unhandled cases (nil, oversized buffers) and correct it.
Simplify the key to a basic type via beam.ParDo before the CoGBK.

Example fix

// before: key of struct without coder support
beam.CoGBK(s, keyedPairs)
// after: encode key as string first
beam.CoGBK(s, beam.ParDo(s, func(k MyKey, v V) (string, V) { return k.String(), v }, keyedPairs))

Defensive patterns

Strategy: try-catch

Validate before calling

// Go: sanity-check that the key encodes before building the pipeline
var buf bytes.Buffer
if err := beam.EncodeElement(context.Background(), key, buf); err != nil { /* switch key type or register coder */ }

Try / catch

if err := beam.Run(...); err != nil {
    var coderErr interface{ Unwrap() error }
    if errors.As(err, &coderErr) { log.Printf("key coder failed: %v", coderErr) }
}

Prevention

Use natively encodable key types (string, int, []byte) for CoGBK keys.
Register coders for custom key types or implement a correct custom coder.
Test Encode/Decode round-trips for custom key coders.
Avoid nil or partially-initialized key structs.

When it happens

Trigger: Calling FinishBundle on a CoGBK node whose pending groups must be reprocessed by window; getGroup fails because the key element cannot be encoded by the pipeline coder for the key PCollection.

Common situations: Using a key type without a registered/derivable coder; a custom coder's Encode method returning an error; keys containing values that violate the coder's assumptions (e.g. nil pointers).

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d912e66504d49d19. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/direct/gbk.go:165

			}
		}
		for k := i; k < j; k++ {
			mergeMap[n.wins[k]] = len(mergedWins)
		}
		mergedWins = append(mergedWins, window.IntervalWindow{Start: mergeStart, End: mergeEnd})
		i = j
	}
	n.wins = mergedWins
	return mergeMap, nil
}

func (n *CoGBK) reprocessByWindow(mergeMap map[typex.Window]int) error {
	newGroups := make(map[string]*group)
	for _, g := range n.m {
		ws := []typex.Window{n.wins[mergeMap[g.key.Windows[0]]]}
		gr, err := n.getGroup(newGroups, &g.key, ws)
		if err != nil {
			return errors.Errorf("failed encoding key for %v: %v", g.key.Elm, err)
		}
		for i, list := range g.values {
			gr.values[i] = append(gr.values[i], list...)
		}
	}
	n.m = newGroups
	return nil
}

func (n *CoGBK) Down(ctx context.Context) error {
	return nil
}

func (n *CoGBK) String() string {
	return fmt.Sprintf("CoGBK. Out:%v", n.Out.ID())
}

// Inject injects the predecessor index into each FullValue, effectively

View on GitHub (pinned to 12126d8942)