apache/beam · error

buildDescriptor: couldn't retrieve coder

Error message

buildDescriptor: couldn't retrieve coder %q for internal pcollection %q: %w

What it means

Internal (stage-produced, stage-consumed) PCollections must have their coders retrievable via retrieveCoders so the bundle descriptor carries complete coder definitions. If retrieval fails — the coder graph is incomplete — prism cannot construct the bundle and returns this error naming the coder ID and internal PCollection.

Solutions

1. Check the wrapped error for which component coder is missing
2. Validate the pipeline's coder component graph before submission
3. Rebuild the pipeline with a supported SDK version
4. Reduce fusion/optimization by restructuring the pipeline to isolate the collection

for _, pid := range internalCols {
    if _, ok := comps.GetCoders()[pcols[pid].GetCoderId()]; !ok {
        return fmt.Errorf("internal pcol %q coder missing", pid)
    }
}

When it happens

Trigger: retrieveCoders(col.GetCoderId(), coders, comps.GetCoders()) errors while iterating stg.internalCols in buildDescriptor.

Common situations: Fused stages whose intermediate PCollections reference partial or unknown coders; optimized graphs where coder components were dropped; version-skewed pipeline protos.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• buildDescriptor: couldn't rewrite coder
• buildDescriptor: failed to handle coder on stage
• buildDescriptor: failed to handle coder on stage
• buildDescriptor: failed to handle coder on stage
• KV coder with more than 2 components

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/91c964e3d49f0163. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/prism/internal/stage.go:647

	inputInfo := engine.PColInfo{
		GlobalID:    stg.primaryInput,
		WindowCoder: winCoder,
		WDec:        wDec,
		WEnc:        wEnc,
		EDec:        ed,
		KeyDec:      kd,
	}

	stg.inputTransformID = stg.ID + "_source"
	transforms[stg.inputTransformID] = sourceTransform(stg.inputTransformID, portFor(wInCid, wk), stg.primaryInput)

	// Update coders for internal collections, and add those collections to the bundle descriptor.
	for _, pid := range stg.internalCols {
		col := clonePColToBundle(pid)
		// Keep the original coder of an internal pcollection without rewriting(LP'ing).
		if err := retrieveCoders(col.GetCoderId(), coders, comps.GetCoders()); err != nil {
			return fmt.Errorf("buildDescriptor: couldn't retrieve coder %q for internal pcollection %q: %w", col.GetCoderId(), pid, err)
		}
	}
	// Add coders for all windowing strategies.
	// TODO: filter PCollections, filter windowing strategies by Pcollections instead.
	for _, ws := range comps.GetWindowingStrategies() {
		lpUnknownCoders(ws.GetWindowCoderId(), coders, comps.GetCoders())
	}

	reconcileCoders(coders, comps.GetCoders())

	var timerServiceDescriptor *pipepb.ApiServiceDescriptor
	if len(stg.hasTimers) > 0 {
		timerServiceDescriptor = &pipepb.ApiServiceDescriptor{
			Url: wk.Endpoint(),
		}
	}

	desc := &fnpb.ProcessBundleDescriptor{

View on GitHub (pinned to 12126d8942)