apache/beam · error

retrieveCoders: couldn't handle component %d %q of %q %v: %w

Error message

retrieveCoders: couldn't handle component %d %q of %q %v:
%w

What it means

retrieveCoders in the prism runner recursively resolves a component coder chain from the pipeline proto, building a bundle of coders keyed by ID. When resolving any component coder ID fails, it wraps the underlying error with the component index, coder ID, and the prototext of the parent coder. This surfaces malformed or unresolvable coder graphs in a submitted pipeline.

Source

Thrown at sdks/go/pkg/beam/runners/prism/internal/coders.go:248

func retrieveCoders(cID string, bundle, base map[string]*pipepb.Coder) error {
	// Look up the canonical location.
	c, ok := base[cID]
	if !ok {
		// We messed up somewhere.
		return fmt.Errorf("retrieveCoders: coder %q not present in base map", cID)
	}

	if _, ok := bundle[cID]; ok {
		return nil
	}
	// Add the original coder to the coders map.
	bundle[cID] = c

	for i, cc := range c.GetComponentCoderIds() {
		// now we need to retrieve the component coders as well
		err := retrieveCoders(cc, bundle, base)
		if err != nil {
			return fmt.Errorf("retrieveCoders: couldn't handle component %d %q of %q %v:\n%w", i, cc, cID, prototext.Format(c), err)
		}
	}

	return nil
}

// reconcileCoders ensures that the bundle coders are primed with initial coders from
// the base pipeline components.
func reconcileCoders(bundle, base map[string]*pipepb.Coder) {
	for {
		var comps []string
		for _, c := range bundle {
			for _, ccid := range c.GetComponentCoderIds() {
				if _, ok := bundle[ccid]; !ok {
					// We don't have the coder yet, so in we go.
					comps = append(comps, ccid)
				}
			}

View on GitHub (pinned to 12126d8942)

Solutions

  1. Check the pipeline proto components: ensure every ComponentCoderId resolves to an existing coder in the same Environment
  2. Upgrade the submitting SDK so its coder encoding matches the prism runner version
  3. Remove or replace custom coder URNs unsupported by prism with standard coders
  4. Read the wrapped inner error (%w) for the root cause at the specific component index

Example fix

// before: pipeline built with dangling component coder ref
// after: validate coder graph before submit
for _, c := range comps.GetCoders() {
  for _, cid := range c.GetComponentCoderIds() {
    if comps.GetCoders()[cid] == nil {
      return fmt.Errorf("dangling component coder ref %q in %q", cid, c.GetUrn())
    }
  }
}
Defensive patterns

Strategy: validation

Validate before calling

comps := pipe.GetComponents()
for _, c := range comps.GetCoders() {
  for _, cid := range c.GetComponentCoderIds() {
    if comps.GetCoders()[cid] == nil {
      return fmt.Errorf("coder %q references missing component coder %q", c.GetSpec().GetUrn(), cid)
    }
  }
}

Prevention

When it happens

Trigger: A pipeline's ComponentCoderIds reference a coder ID absent from the pipeline components, a component coder is itself malformed, or a custom coder URN unsupported by prism appears in the chain.

Common situations: Submitting pipelines from SDKs with coder encoding mismatches (version skew between SDK and prism), hand-crafted or serialized pipeline protos with dangling coder references, or custom coders prism cannot handle.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b8630f4cb13e75dc. Report an issue: GitHub.