apache/beam · error

invalid transform payload for %v

Error message

invalid transform payload for %v

What it means

After extracting the DoFn/CombineFn payload string, makeLink decodes it as a base64-encoded v1pb.TransformPayload. If DecodeBase64/proto decoding fails, this wrapped error reports the transform that could not be decoded. This is the legacy Dataflow transform-payload path.

Source

Thrown at sdks/go/pkg/beam/core/runtime/exec/translate.go:505

		case urnPerKeyCombinePre, urnPerKeyCombineMerge, urnPerKeyCombineExtract, urnPerKeyCombineConvert:
			var cmb pipepb.CombinePayload
			if err := proto.Unmarshal(payload, &cmb); err != nil {
				return nil, errors.Wrapf(err, "invalid CombinePayload payload for %v", transform)
			}
			data = string(cmb.GetCombineFn().GetPayload())
		default:
			// TODO(herohde) 12/4/2017: we see DoFns directly with Dataflow. Handle that
			// case here, for now, so that the harness can use this logic.

			data = string(payload)
		}

		// TODO(herohde) 1/28/2018: Once Dataflow's fully off the old way,
		// we can simply switch on the ParDo DoFn URN directly.

		var tp v1pb.TransformPayload
		if err := protox.DecodeBase64(data, &tp); err != nil {
			return nil, errors.Wrapf(err, "invalid transform payload for %v", transform)
		}

		switch tpUrn := tp.GetUrn(); tpUrn {
		case graphx.URNDoFn:
			op, fn, _, in, _, err := graphx.DecodeMultiEdge(tp.GetEdge())
			if err != nil {
				return nil, err
			}

			switch op {
			case graph.ParDo:
				dofn, err := graph.AsDoFn(fn, graph.MainUnknown)
				if err != nil {
					return nil, err
				}
				switch urn {
				case urnPairWithRestriction:
					u = &PairWithRestriction{UID: b.idgen.New(), Fn: dofn, Out: out[0]}

View on GitHub (pinned to 12126d8942)

Solutions

1. Check that the transform URN is one makeLink handles, so the payload is decoded by the right branch rather than the legacy TransformPayload path
2. Ensure the payload string is proper base64 (protox.DecodeBase64) encoding of a TransformPayload
3. Regenerate the pipeline with a current SDK so modern payload encodings are used

// Check data is decodable base64 TransformPayload before plan build
var tp v1pb.TransformPayload
if err := protox.DecodeBase64(data, &tp); err != nil {
    return fmt.Errorf("undecodable transform payload: %w", err)
}

if err := exec.UnmarshalPlan(desc); err != nil {
    if strings.Contains(err.Error(), "invalid transform payload") {
        return fmt.Errorf("legacy transform payload undecodable for transform: %w", err)
    }
    return err
}

When it happens

Trigger: Transform payload data string is not valid base64 or not a valid TransformPayload proto — e.g. empty data from an unhandled URN in the default branch, or legacy payloads — during makeLink.

Common situations: Dataflow-style harness paths where DoFns appear directly; pipelines built by older SDK encodings; unhandled transform URNs falling into the default branch with unusable data.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• empty type
• invalid ParDo payload for %v
• invalid CombinePayload payload for %v
• invalid transform payload %v for %v
• invalid WindowInto payload for %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/31d0a688ff32b433. Report an issue: GitHub.