apache/beam · error

could not unmarshal CoderRef from %v, failed to decode urn-l

Error message

could not unmarshal CoderRef from %v, failed to decode urn-less coder's payload "%v"

What it means

A coder with an empty URN (urn-less CoderRef, often seen from legacy Dataflow) carries a JSON payload that failed to json.Unmarshal into a CoderRef struct. The payload is therefore not a valid serialized coder reference.

Source

Thrown at sdks/go/pkg/beam/core/runtime/graphx/coder.go:334

		if urn == urnWindowedValueCoder {
			return wvc, nil
		}
		wvc.Kind = coder.ParamWindowedValue
		wvc.Window.Payload = string(c.GetSpec().GetPayload())
		return wvc, nil

	case streamType:
		return nil, errors.Errorf("could not unmarshal stream type coder from %v, stream must be pair value", c)

	case "":
		// TODO(herohde) 11/27/2017: we still see CoderRefs from Dataflow. Handle that
		// case here, for now, so that the harness can use this logic.

		payload := c.GetSpec().GetPayload()

		var ref CoderRef
		if err := json.Unmarshal(payload, &ref); err != nil {
			return nil, errors.Wrapf(err, "could not unmarshal CoderRef from %v, failed to decode urn-less coder's payload \"%v\"", c, string(payload))
		}
		c, err := DecodeCoderRef(&ref)
		if err != nil {
			return nil, errors.Wrapf(err, "could not unmarshal CoderRef from %v, failed to decode CoderRef \"%v\"", c, string(payload))
		}
		return c, nil

	case urnIterableCoder, urnStateBackedIterableCoder:
		if len(components) != 1 {
			return nil, errors.Errorf("could not unmarshal iterable coder from %v, expected one component but got %d", c, len(components))
		}
		elm, err := b.Coder(components[0])
		if err != nil {
			return nil, err
		}
		return coder.NewI(elm), nil
	case urnTimerCoder:
		if len(components) != 2 {

View on GitHub (pinned to 12126d8942)

Solutions

1. Log/inspect the raw payload to see why JSON parsing fails (truncation, wrong schema)
2. Re-serialize the pipeline with a compatible Go SDK version
3. Validate the pipeline JSON before submission
4. If legacy Dataflow, migrate to the portability/model pipeline format

Example fix

// before: payload with wrong JSON shape
// after: ensure payload marshals graphx.CoderRef, e.g. {"urn":"beam:coder:kv:v1","components":[...]}

var probe map[string]any
if err := json.Unmarshal(payload, &probe); err != nil {
    return fmt.Errorf("urn-less coder payload is not valid JSON: %w", err)
}
if _, ok := probe["urn"]; !ok {
    return errors.New("urn-less coder payload missing 'urn' field")
}

c, err := graphx.Coder(model, ref)
if err != nil {
    return fmt.Errorf("decoder could not parse CoderRef payload %q: %w", payload, err)
}

When it happens

Trigger: Decoding a coder whose spec has no URN and whose payload is not valid CoderRef JSON (truncated, wrong fields, empty).

Common situations: Legacy Dataflow CoderRefs in pipeline graphs; corrupted pipeline files; payloads written by incompatible SDK versions.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• GroupIntoBatches: key coder for type %v is not deterministic
• invalid coder
• not a float type: %v
• not a signed integer type: %v
• not a unsigned integer type: %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4ca4371b601fe3b7. Report an issue: GitHub.