apache/beam · critical

retrieveCoders: coder %q not present in base map

Error message

retrieveCoders: coder %q not present in base map

What it means

retrieveCoders collects all coders required for a bundle descriptor from the base map into the bundle map. This error means a required coder ID is not present in the base coder map, so the descriptor for execution cannot be completed — the pipeline's coder graph has a dangling reference.

Source

Thrown at sdks/go/pkg/beam/runners/prism/internal/coders.go:235

		Spec: &pipepb.FunctionSpec{
			Urn: urns.CoderLengthPrefix,
		},
		ComponentCoderIds: []string{cID},
	}
	bundle[lpcID] = lpc
	return lpcID, nil
}

// retrieveCoders recursively ensures that the coder along with all its direct
// and indirect component coders, are present in the `bundle` map.
// If a coder is already in `bundle`, it's skipped. Returns an error if any
// required coder ID is not found.
func retrieveCoders(cID string, bundle, base map[string]*pipepb.Coder) error {
	// Look up the canonical location.
	c, ok := base[cID]
	if !ok {
		// We messed up somewhere.
		return fmt.Errorf("retrieveCoders: coder %q not present in base map", cID)
	}

	if _, ok := bundle[cID]; ok {
		return nil
	}
	// Add the original coder to the coders map.
	bundle[cID] = c

	for i, cc := range c.GetComponentCoderIds() {
		// now we need to retrieve the component coders as well
		err := retrieveCoders(cc, bundle, base)
		if err != nil {
			return fmt.Errorf("retrieveCoders: couldn't handle component %d %q of %q %v:\n%w", i, cc, cID, prototext.Format(c), err)
		}
	}

	return nil
}

View on GitHub (pinned to 12126d8942)

Solutions

  1. Ensure every referenced coder ID (including window and component coders) is defined in the pipeline components before submission
  2. Check Beam SDK and prism runner version alignment and upgrade to a matching release
  3. If reproducible with a standard pipeline, file a Beam issue with the pipeline proto for diagnosis
Defensive patterns

Strategy: validation

Validate before calling

if _, ok := base[cID]; !ok {
    return fmt.Errorf("required coder %q not defined", cID)
}

Try / catch

if err := retrieveCoders(cID, bundle, base); err != nil {
    return fmt.Errorf("descriptor build failed: %w", err)
}

Prevention

When it happens

Trigger: buildDescriptor (or recursive retrieveCoders on component coders) encounters a coder ID referenced by a PCollection, window coder, or composite that was never defined in the base components.

Common situations: Pipeline protos with dropped/trimmed coders still referenced; SDK/runner version mismatches producing coder IDs unknown to the runner; custom coders added to PCollections without registering component coders.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/aa36d1d710b1840a. Report an issue: GitHub.