apache/beam · error

invalid ParDo payload for %v

Error message

invalid ParDo payload for %v

What it means

makeLink decodes the ParDoPayload proto from a ParDo-family transform's spec payload. If proto.Unmarshal fails (malformed/empty payload bytes), the error is wrapped with the transform name and returned, aborting plan translation.

Source

Thrown at sdks/go/pkg/beam/core/runtime/exec/translate.go:481

		urnPerKeyCombineExtract,
		urnPerKeyCombineConvert,
		urnPairWithRestriction,
		urnSplitAndSizeRestrictions,
		urnProcessSizedElementsAndRestrictions,
		urnTruncateSizedRestrictions:
		var data string
		var sides map[string]*pipepb.SideInput
		var userState map[string]*pipepb.StateSpec
		var userTimers map[string]*pipepb.TimerFamilySpec
		switch urn {
		case graphx.URNParDo,
			urnPairWithRestriction,
			urnSplitAndSizeRestrictions,
			urnProcessSizedElementsAndRestrictions,
			urnTruncateSizedRestrictions:
			var pardo pipepb.ParDoPayload
			if err := proto.Unmarshal(payload, &pardo); err != nil {
				return nil, errors.Wrapf(err, "invalid ParDo payload for %v", transform)
			}
			data = string(pardo.GetDoFn().GetPayload())
			sides = pardo.GetSideInputs()
			userState = pardo.GetStateSpecs()
			userTimers = pardo.GetTimerFamilySpecs()
		case urnPerKeyCombinePre, urnPerKeyCombineMerge, urnPerKeyCombineExtract, urnPerKeyCombineConvert:
			var cmb pipepb.CombinePayload
			if err := proto.Unmarshal(payload, &cmb); err != nil {
				return nil, errors.Wrapf(err, "invalid CombinePayload payload for %v", transform)
			}
			data = string(cmb.GetCombineFn().GetPayload())
		default:
			// TODO(herohde) 12/4/2017: we see DoFns directly with Dataflow. Handle that
			// case here, for now, so that the harness can use this logic.

			data = string(payload)
		}

View on GitHub (pinned to 12126d8942)

Solutions

1. Ensure the transform spec payload is a valid serialized ParDoPayload (build pipelines through beam.ParDo/graphx encoding, not manual protos)
2. Log/inspect the payload bytes and verify they decode with pipepb.ParDoPayload in a test
3. Upgrade SDK versions if the runner emits a different payload encoding

// Verify payload decodes before plan translation
var pardo pipepb.ParDoPayload
if err := proto.Unmarshal(spec.GetPayload(), &pardo); err != nil {
    return fmt.Errorf("bad ParDoPayload: %w", err)
}

if err := exec.UnmarshalPlan(desc); err != nil {
    if strings.Contains(err.Error(), "invalid ParDo payload") {
        return fmt.Errorf("pipeline graph malformed at ParDo transform: %w", err)
    }
    return err
}

When it happens

Trigger: A transform with URN urnParDo (or SDF-related URNs like pair-with-restriction, split-and-size-restrictions, process-sized-elements) whose spec payload is not a valid serialized pipepb.ParDoPayload, during makeLink.

Common situations: Pipeline graphs built outside the standard Beam pipeline-construction path (hand-crafted protos, tests); cross-SDK graph conversions that don't populate ParDo payloads; corrupted payloads from legacy Dataflow-style DoFn handling.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• empty type
• invalid CombinePayload payload for %v
• invalid transform payload for %v
• invalid transform payload %v for %v
• invalid WindowInto payload for %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c1c2e41cf55e9bfa. Report an issue: GitHub.