apache/beam · error

failed to add output coder to coder registry

Error message

failed to add output coder to coder registry: %v

What it means

Specifically for Python-origin ExternalTransforms (URN of the form beam:transform:...:python), expandCrossLanguage registers each output PCollection's coder in the coder registry. If m.coders.Add fails (coder cannot be serialized/registered), the error is wrapped as 'failed to add output coder to coder registry'. The coder registry ran out of IDs or rejected the coder payload.

Solutions

1. Inspect the wrapped error to see which coder failed to register
2. Ensure output coders are standard/serializable (e.g. use coders the Go SDK supports across languages)
3. Check URN format: this path only triggers when names[2]=="python"; a malformed URN could route to the wrong branch
4. Upgrade the Go SDK so newer coder payload formats are understood

Example fix

null

// ensure output coders are standard types before external transform
for _, out := range outputs {
    if !isStandardCoder(out.Coder) {
        return fmt.Errorf("output coder %T not cross-language safe", out.Coder)
    }
}

err := beam.Run(ctx, pr)
if err != nil && strings.Contains(err.Error(), "failed to add output coder to coder registry") {
    log.Printf("output coder not registerable: %v", err)
    return fmt.Errorf("use standard output coders for python external transforms: %w", err)
}

When it happens

Trigger: An ExternalTransform with a python URN whose output coder cannot be added to the marshaller's coder registry — e.g. a coder whose payload marshalling fails or an unsupported custom coder crossing the language boundary.

Common situations: Python cross-language transforms producing PCollections with custom coders the Go side cannot represent; version drift between the Python transform's coder payload and the Go SDK's coder support.

Related errors

• attempted to add namespace to missing coder id
• attempted to add namespace to missing windowing strategy id
• failed to expand cross language transform for edge
• invalid scope
• malformed components;

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/08ffe027a1440982. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/graphx/translate.go:774

		Payload: edge.External.Payload,
	}

	transform := &pipepb.PTransform{
		UniqueName:    namedEdge.Name,
		Spec:          spec,
		Inputs:        inputs,
		EnvironmentId: m.addDefaultEnv(),
	}

	// Add the coders for output in the marshaller even if expanded is nil
	// for output coder field in expansion request.
	// We need this specifically for Python External Transforms.
	names := strings.Split(spec.Urn, ":")
	if len(names) > 2 && names[2] == "python" {
		for _, out := range edge.Output {
			id, err := m.coders.Add(out.To.Coder)
			if err != nil {
				return "", errors.Wrapf(err, "failed to add output coder to coder registry: %v", m.coders)
			}
			out.To.Coder.ID = id
		}
	}

	if edge.External.Expanded != nil {
		// Outputs need to temporarily match format of unnamed Go SDK Nodes.
		// After the initial pipeline is constructed, these will be used to correctly
		// map consumers of these outputs to the expanded transform's outputs.
		outputs := make(map[string]string)
		for i, out := range edge.Output {
			if _, err := m.addNode(out.To); err != nil {
				return "", errors.Wrapf(err, "failed to expand cross language transform for edge: %v", namedEdge)
			}
			outputs[fmt.Sprintf("i%v", i)] = nodeID(out.To)
		}
		transform.Outputs = outputs
		environment, err := ExpandedTransform(edge.External.Expanded)

View on GitHub (pinned to 12126d8942)