apache/beam · error

unknown coder id during reconciliation

Error message

unknown coder id during reconciliation: %v

What it means

During coder reconciliation prism builds a bundle-scoped coder map from the base pipeline coders. If a component coder ID referenced by a composite coder is absent from the base map, the invariants of the pipeline proto are violated and it panics.

Solutions

1. Verify the pipeline proto contains all referenced coder components (dump the model pipeline)
2. Upgrade/downgrade the Beam SDK and prism runner to matching versions
3. Check custom transform/coder registration so every composite coder's component IDs are registered
4. File a Beam bug with the pipeline if this occurs from standard SDK output

Example fix

// before
// coder 'c5' referenced but missing from components
// after
// ensure every GetComponentCoderIds entry exists: add coder c5 to pipeline.Components before submit

for _, c := range pipeline.GetComponents().GetCoders() {
    for _, id := range c.GetComponentCoderIds() {
        if !coderIDs[id] { return fmt.Errorf("dangling coder reference %q", id) }
    }
}

defer func() {
    if r := recover(); r != nil && strings.Contains(fmt.Sprint(r), "unknown coder id during reconciliation") {
        err = fmt.Errorf("pipeline proto has unreferenced coder: %v", r)
    }
}()

When it happens

Trigger: A pipeline whose components reference a coder ID never registered in the components list — typically a graph serialization bug or a hand-edited/stale model pipeline sent to prism.

Common situations: Runner/SDK version mismatches where the SDK emits coder references the submitted components don't contain, or custom pipeline transform builders producing dangling coder IDs.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

• 'Cannot provide coder for
• Cannot provide SerializableCoder because
• Forbidden IOException when writing to OutputStream
• Iterable coder must have only one component
• Java Serialization may be non-deterministic.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6130f34be1a19ba9. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/prism/internal/coders.go:274

// the base pipeline components.
func reconcileCoders(bundle, base map[string]*pipepb.Coder) {
	for {
		var comps []string
		for _, c := range bundle {
			for _, ccid := range c.GetComponentCoderIds() {
				if _, ok := bundle[ccid]; !ok {
					// We don't have the coder yet, so in we go.
					comps = append(comps, ccid)
				}
			}
		}
		if len(comps) == 0 {
			return
		}
		for _, ccid := range comps {
			c, ok := base[ccid]
			if !ok {
				panic(fmt.Sprintf("unknown coder id during reconciliation: %v", ccid))
			}
			bundle[ccid] = c
		}
	}
}

// pullDecoder return a function that will extract the bytes
// for the associated coder. Uses a buffer and a TeeReader to extract the original
// bytes from when decoding elements.
func pullDecoder(c *pipepb.Coder, coders map[string]*pipepb.Coder) func(io.Reader) []byte {
	dec := pullDecoderNoAlloc(c, coders)
	return func(r io.Reader) []byte {
		var buf bytes.Buffer
		tr := io.TeeReader(r, &buf)
		dec(tr)
		return buf.Bytes()
	}
}

View on GitHub (pinned to 12126d8942)