apache/beam · error

buildDescriptor: couldn't rewrite coder

Error message

buildDescriptor: couldn't rewrite coder %q for primary input pcollection %q: %w

What it means

The stage's primary input PCollection coder is rewritten (LP'd) to concrete coders before building the bundle descriptor. If lpUnknownCoders fails, prism cannot encode/decode the stage's main input and returns this error naming the original coder ID and PCollection.

Solutions

1. Inspect the wrapped error to identify the failing coder URN or component
2. Ensure the input PCollection's coder is fully specified in pipeline components
3. Regenerate the pipeline with a compatible SDK version
4. Avoid hand-editing pipeline protos; use the SDK's coder registration APIs

Example fix

// before: hand-built proto references coder "unknown1" absent from components
// after: register the coder in the pipeline graph
beam.RegisterCoder(reflect.TypeOf(MyType{}), enc, dec)

if _, ok := comps.GetCoders()[inputPcol.GetCoderId()]; !ok {
    return fmt.Errorf("primary input coder %q not in components", inputPcol.GetCoderId())
}

When it happens

Trigger: lpUnknownCoders(col.GetCoderId(), coders, comps.GetCoders()) returns an error for stg.primaryInput during buildDescriptor.

Common situations: Primary inputs produced by external transforms or cross-language expansion with unknown coders; pipelines with unregistered custom coder URNs; SDK/runner version skew.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• buildDescriptor: couldn't retrieve coder
• buildDescriptor: failed to handle coder on stage
• buildDescriptor: failed to handle coder on stage
• buildDescriptor: failed to handle coder on stage
• KV coder with more than 2 components

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/914d5ef7558f0d06. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/prism/internal/stage.go:615

		}
		prepSide, err := handleSideInput(si, comps, transforms, pcollections, coders, em)
		if err != nil {
			slog.Error("buildDescriptor: handleSideInputs", "error", err, slog.String("transformID", si.Transform))
			return err
		}
		prepareSides = append(prepareSides, prepSide)
	}

	// Finally, the parallel input, which is it's own special snowflake, that needs a datasource.
	// This id is directly used for the source, but this also copies
	// coders used by side inputs to the coders map for the bundle, so
	// needs to be run for every ID.

	col := clonePColToBundle(stg.primaryInput)
	if newCID, err := lpUnknownCoders(col.GetCoderId(), coders, comps.GetCoders()); err == nil && col.GetCoderId() != newCID {
		col.CoderId = newCID
	} else if err != nil {
		return fmt.Errorf("buildDescriptor: couldn't rewrite coder %q for primary input pcollection %q: %w", col.GetCoderId(), stg.primaryInput, err)
	}

	wInCid, err := makeWindowedValueCoder(stg.primaryInput, comps, coders)
	if err != nil {
		return fmt.Errorf("buildDescriptor: failed to handle coder on stage %v for primary input, pcol %q %v:\n%w\n%v", stg.ID, stg.primaryInput, prototext.Format(col), err, stg.transforms)
	}
	ed := collectionPullDecoder(col.GetCoderId(), coders, comps)
	winCoder, wDec, wEnc := getWindowValueCoders(comps, col, coders)

	var kd func(io.Reader) []byte
	if kcid, ok := extractKVCoderID(col.GetCoderId(), coders); ok {
		kd = collectionPullDecoder(kcid, coders, comps)
	}

	inputInfo := engine.PColInfo{
		GlobalID:    stg.primaryInput,
		WindowCoder: winCoder,
		WDec:        wDec,

View on GitHub (pinned to 12126d8942)