apache/beam · critical

lpUnknownCoders: coder %q not present in base map

Error message

lpUnknownCoders: coder %q not present in base map

What it means

lpUnknownCoders looks up each coder ID in the pipeline's base coder map to decide whether it must be length-prefixed for the runner. This error means a coder ID was referenced but never defined in the base map — an internal consistency violation in the pipeline's coder graph, which the code comments as "we messed up somewhere."

Source

Thrown at sdks/go/pkg/beam/runners/prism/internal/coders.go:149

// lpUnknownCoders takes a coder, and populates coders with any new coders
// coders that the runner needs to be safe, and speedy.
// It returns either the passed in coder id, or the new safe coder id.
func lpUnknownCoders(cID string, bundle, base map[string]*pipepb.Coder) (string, error) {
	// First check if we've already added the LP version of this coder to coders already.
	lpcID := cID + "_lp"
	// Check if we've done this one before.
	if _, ok := bundle[lpcID]; ok {
		return lpcID, nil
	}
	// All coders in the coders map have been processed.
	if _, ok := bundle[cID]; ok {
		return cID, nil
	}
	// Look up the canonical location.
	c, ok := base[cID]
	if !ok {
		// We messed up somewhere.
		return "", fmt.Errorf("lpUnknownCoders: coder %q not present in base map", cID)
	}
	// Add the original coder to the coders map.
	bundle[cID] = c
	// If we don't know this coder, we must LP it, and we return the LP'd version.
	leaf := isLeafCoder(c)
	knownComposite := isKnownCompositeCoder(c)
	if !leaf && !knownComposite {
		lpc := &pipepb.Coder{
			Spec: &pipepb.FunctionSpec{
				Urn: urns.CoderLengthPrefix,
			},
			ComponentCoderIds: []string{cID},
		}
		bundle[lpcID] = lpc
		return lpcID, nil
	}
	// If it is a leaf, move its components (if any) to the coders map.
	if leaf {

View on GitHub (pinned to 12126d8942)

Solutions

  1. Dump the pipeline proto coders map and find which builder step omitted the referenced coder ID
  2. Check for recent changes to pipeline construction/trimming that could drop coders still in use
  3. Report to beam-dev or file a Beam issue with the pipeline proto if it comes from a standard SDK pipeline
Defensive patterns

Strategy: validation

Validate before calling

if _, ok := comps.GetCoders()[coderID]; !ok {
    return fmt.Errorf("coder %q referenced but not defined", coderID)
}

Try / catch

if _, err := lpUnknownCoders(cID, bundle, base); err != nil {
    return fmt.Errorf("coder resolution failed: %w", err)
}

Prevention

When it happens

Trigger: Any of lpUnknownCoders' callers (makeWindowedValueCoder, recursive component traversal, collectionPullDecoder, getWindowValueCoders, handleTestStream) pass a coder ID absent from the base components' coders map.

Common situations: A PCollection or composite coder references a component coder ID that was never added when the pipeline proto was built; trimming logic dropped a coder still referenced; TestStream constructions with stale coder IDs.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8d04aa728c3ac7a6. Report an issue: GitHub.