apache/beam · error

failed to marshal custom coder %v

Error message

failed to marshal custom coder %v

What it means

While marshalling a coder with a Custom type, graphx failed to encode the custom coder reference into base64 proto payload after the spec was built. This wraps an underlying error from protox.EncodeBase64 and indicates the custom coder spec could not be serialized for embedding in the pipeline model.

Source

Thrown at sdks/go/pkg/beam/core/runtime/graphx/coder.go:466

	return &CoderMarshaller{
		coders:   make(map[string]*pipepb.Coder),
		coder2id: make(map[string]string),
	}
}

// Add adds the given coder to the set and returns its id. Idempotent.
func (b *CoderMarshaller) Add(c *coder.Coder) (string, error) {
	switch c.Kind {
	case coder.Custom:
		ref, err := encodeCustomCoder(c.Custom)
		if err != nil {
			return "", errors.SetTopLevelMsgf(err, "failed to encode custom coder %s for TypeName %s. "+
				"Make sure the type was registered before calling beam.Init. For example: "+
				"beam.RegisterType(reflect.TypeOf((*TypeName)(nil)).Elem()). Some types, like maps, slices, arrays, channels, and functions cannot be registered as types.", c, c.Custom.Type)
		}
		data, err := protox.EncodeBase64(ref)
		if err != nil {
			return "", errors.Wrapf(err, "failed to marshal custom coder %v", c)
		}
		inner := b.internCoder(&pipepb.Coder{
			Spec: &pipepb.FunctionSpec{
				Urn:     urnCustomCoder,
				Payload: []byte(data),
			},
		})
		return b.internBuiltInCoder(urnLengthPrefixCoder, inner), nil

	case coder.KV:
		comp, err := b.AddMulti(c.Components)
		if err != nil {
			return "", errors.Wrapf(err, "failed to marshal KV coder %v", c)
		}
		return b.internBuiltInCoder(urnKVCoder, comp...), nil

	case coder.Nullable:
		comp, err := b.AddMulti(c.Components)

View on GitHub (pinned to 12126d8942)

Solutions

1. Register the custom type before beam.Init: beam.RegisterType(reflect.TypeOf((*T)(nil)).Elem())
2. Read the inner/top-level message: it names the TypeName that failed encoding
3. Replace the unserializable type with a serializable struct or use a custom coder implementation
4. Upgrade the SDK if the type should be encodable

Example fix

// before
func init() { /* nothing registered */ }
// after
func init() {
  beam.RegisterType(reflect.TypeOf((*MyCustomType)(nil)).Elem())
}

// Ensure the custom type is registered before pipeline marshalling
var _ = func() bool {
    t := reflect.TypeOf((*MyType)(nil)).Elem()
    return graphx.GetCustomCoderRef(t) != nil // or check via beam.RegisterType in init()
}

// This error already embeds a top-level hint; catch and surface it
if err != nil && strings.Contains(err.Error(), "failed to encode custom coder") {
    log.Fatalf("register type: %v", err)
}

When it happens

Trigger: Calling CoderUnmarshaller/CoderBuilder Add with a coder.Custom whose underlying registration payload fails base64/proto encoding, typically because the type's coder ref couldn't be encoded.

Common situations: Registering exotic types (maps, slices, channels, functions) that cannot get stable coder refs; missing beam.RegisterType registration causing an unencodable ref.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• varint too long
• custom coders must be length prefixed: %+v
• bad encoding function
• bad decoding function
• value %v not encodable with %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0938a8389a4c4617. Report an issue: GitHub.