apache/beam · error

invalid transform payload %v for %v

Error message

invalid transform payload %v for %v

What it means

During pipeline decoding in makeLink, when a ParDo has a CombiningSpec user state, the combine fn's base64-encoded TransformPayload is decoded. If protox.DecodeBase64 fails, the translation aborts wrapped in this error, meaning the runner sent a malformed or unreadable combine-fn payload for a stateful DoFn.

Source

Thrown at sdks/go/pkg/beam/core/runtime/exec/translate.go:550

					input := unmarshalKeyedValues(transform.GetInputs())

					if len(userState) > 0 {
						stateIDToCoder := make(map[string]*coder.Coder)
						stateIDToKeyCoder := make(map[string]*coder.Coder)
						stateIDToCombineFn := make(map[string]*graph.CombineFn)
						for key, spec := range userState {
							var cID string
							var kcID string
							if rmw := spec.GetReadModifyWriteSpec(); rmw != nil {
								cID = rmw.CoderId
							} else if bs := spec.GetBagSpec(); bs != nil {
								cID = bs.ElementCoderId
							} else if cs := spec.GetCombiningSpec(); cs != nil {
								cID = cs.AccumulatorCoderId
								cmbData := string(cs.GetCombineFn().GetPayload())
								var cmbTp v1pb.TransformPayload
								if err := protox.DecodeBase64(cmbData, &cmbTp); err != nil {
									return nil, errors.Wrapf(err, "invalid transform payload %v for %v", cmbData, transform)
								}
								_, fn, _, _, _, err := graphx.DecodeMultiEdge(cmbTp.GetEdge())
								if err != nil {
									return nil, err
								}
								cfn, err := graph.AsCombineFn(fn)
								if err != nil {
									return nil, err
								}
								stateIDToCombineFn[key] = cfn
							} else if ms := spec.GetMapSpec(); ms != nil {
								cID = ms.ValueCoderId
								kcID = ms.KeyCoderId
							} else if ss := spec.GetSetSpec(); ss != nil {
								kcID = ss.ElementCoderId
							} else if ols := spec.GetOrderedListSpec(); ols != nil {
								cID = ols.ElementCoderId
							} else {

View on GitHub (pinned to 12126d8942)

Solutions

1. Pin the Go Beam SDK and runner/container to the same version and redeploy
2. Regenerate/rebuild the pipeline so the combine payload is re-encoded, and resubmit the job
3. Decode the cmbData base64 offline to confirm it parses as v1pb.TransformPayload before submitting
4. Update custom transform code that manually writes CombiningSpec payloads to use the official graphx encoding

Example fix

// Hand-built state spec with raw payload
spec.CombiningSpec.CombineFn.Payload = []byte(myString)
// after: encode properly
var cmb v1pb.TransformPayload
b, _ := proto.Marshal(&cmb)
spec.CombiningSpec.CombineFn.Payload = []byte(base64.StdEncoding.EncodeToString(b))

var tp v1pb.TransformPayload
if _, err := base64.StdEncoding.DecodeString(string(cs.GetCombineFn().GetPayload())); err != nil {
	return fmt.Errorf("combine payload is not valid base64: %w", err)
}
if err := protox.DecodeBase64(string(cs.GetCombineFn().GetPayload()), &tp); err != nil {
	return fmt.Errorf("combine payload does not decode: %w", err)
}

null

null

When it happens

Trigger: A combining user state (beam.CombineContext/PerKey state) whose CombineFn payload is not valid base64-encoded TransformPayload protobuf — e.g. corrupt job submission, cross-version runner/Fn Harness mismatch, or a custom transform writing a hand-built state spec.

Common situations: Runner and SDK harness from different Beam versions where the TransformPayload encoding changed; manually constructed pipeline protobufs; corrupted/stale worker caches during container upgrades.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• empty type
• expected one output from DataSource, got %v
• pcollection %v not found
• windowing strategy %v not found
• invalid ParDo payload for %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7daec20ebfaa3137. Report an issue: GitHub.