apache/beam · error

custom coders must be length prefixed: %+v

Error message

custom coders must be length prefixed: %+v

What it means

graphx.DecodeCoderRef converts a serialized Dataflow CoderRef into a beam coder. Only a limited set of coder kinds is supported; when the CoderRef kind does not match any known case (including the custom-coder path that requires a length-prefixed payload), it fails. Custom coders must carry a length-prefixed payload to be decoded.

Source

Thrown at sdks/go/pkg/beam/core/runtime/graphx/dataflow.go:357

		}

		t := typex.New(reflect.SliceOf(inner.T.Type()), inner.T)
		return &coder.Coder{Kind: coder.Iterable, T: t, Components: []*coder.Coder{inner}}, nil

	case rowType:
		subC := c.Components[0]
		schm := &pipepb.Schema{}
		if err := protox.DecodeBase64(subC.Type, schm); err != nil {
			return nil, err
		}
		t, err := schema.ToType(schm)
		if err != nil {
			return nil, err
		}
		return &coder.Coder{Kind: coder.Row, T: typex.New(t)}, nil

	default:
		return nil, errors.Errorf("custom coders must be length prefixed: %+v", c)
	}
}

func decodeDataflowCustomCoder(payload string) (*coder.Coder, error) {
	var ref v1pb.CustomCoder
	if err := protox.DecodeBase64(payload, &ref); err != nil {
		return nil, errors.Wrapf(err, "base64 decode for %v failed", payload)
	}
	custom, err := decodeCustomCoder(&ref)
	if err != nil {
		return nil, err
	}
	t := typex.New(custom.Type)
	return &coder.Coder{Kind: coder.Custom, T: t, Custom: custom}, nil
}

func isCoGBKList(ref *CoderRef) ([]*CoderRef, bool) {
	if ref.Type != lengthPrefixType {

View on GitHub (pinned to 12126d8942)

Solutions

1. Upgrade or align the Beam Go SDK version used to serialize and deserialize the pipeline so both sides support the same coder kinds.
2. Ensure custom coders are wrapped in a length-prefixed coder (CoderFromType / beam.EncodedCustomCoder) before serialization.
3. If the type can be represented natively, use a standard coder (RowCoder) instead of a custom coder.

Example fix

// before: raw custom coder, not length prefixed
// after
beam.CoderFromType(s, reflect.TypeOf(myType{})) // uses Row/length-prefixed representation
// or ensure custom coder registration:
coder.RegisterCustomCoder("mycoder", encode, decode)

// Before submitting: verify coder ref decodes
if _, err := graphx.DecodeCoderRef(ref); err != nil {
    return fmt.Errorf("unsupported coder ref kind %v: %w", ref.Kind, err)
}

if err != nil {
    var unsupported bool
    if strings.Contains(err.Error(), "custom coders must be length prefixed") { unsupported = true }
    // fall back to RowCoder re-serialization
}

When it happens

Trigger: Decoding a CoderRef whose Kind is unrecognized, or a custom coder whose payload was not encoded with the required length-prefixing, via DecodeCoderRef / DecodeCoderRefs / DecodeCoder (typically while deserializing a saved model or pipeline for execution on a runner).

Common situations: Pipelines serialized by a different/newer Beam version introducing a coder kind this version doesn't know; hand-crafted or runner-rewritten coder refs; custom coders registered on one side of a job-submission boundary but not round-tripped correctly.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• varint too long
• failed to marshal custom coder %v
• bad encoding function
• bad decoding function
• value %v not encodable with %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/48d6331e1ab10933. Report an issue: GitHub.