apache/beam · error

unable to decode ParDoPayload for %v

Error message

unable to decode ParDoPayload for %v

What it means

finalizeStage inspects each ParDo-family transform in a stage and must decode its ParDoPayload; this error is returned when proto unmarshaling of the transform spec payload fails. Prism needs the payload to know about finalization requests, timers, and state specs.

Source

Thrown at sdks/go/pkg/beam/runners/prism/internal/preprocess.go:469

	var sideInputs []engine.LinkID
	inputs := map[string]bool{}
	for pid, plinks := range stageFacts.PcolConsumers {
		// Check if this PCollection is generated in this bundle.
		if _, ok := stageFacts.PcolProducers[pid]; ok {
			// It is, so we will ignore for now.
			continue
		}
		// Add this collection to our input set.
		inputs[pid] = true
		for _, link := range plinks {
			t := comps.GetTransforms()[link.Transform]

			var sis map[string]*pipepb.SideInput
			switch t.GetSpec().GetUrn() {
			case urns.TransformParDo, urns.TransformProcessSizedElements, urns.TransformPairWithRestriction, urns.TransformSplitAndSize, urns.TransformTruncate:
				pardo := &pipepb.ParDoPayload{}
				if err := (proto.UnmarshalOptions{}).Unmarshal(t.GetSpec().GetPayload(), pardo); err != nil {
					return fmt.Errorf("unable to decode ParDoPayload for %v", link.Transform)
				}
				if pardo.GetRequestsFinalization() {
					stg.finalize = true
				}
				if len(pardo.GetTimerFamilySpecs())+len(pardo.GetStateSpecs())+len(pardo.GetOnWindowExpirationTimerFamilySpec()) > 0 {
					stg.stateful = true
				}
				if pardo.GetOnWindowExpirationTimerFamilySpec() != "" {
					stg.onWindowExpiration = engine.StaticTimerID{TransformID: link.Transform, TimerFamily: pardo.GetOnWindowExpirationTimerFamilySpec()}
				}
				sis = pardo.GetSideInputs()
			}
			if _, ok := sis[link.Local]; ok {
				sideInputs = append(sideInputs, engine.LinkID{Transform: link.Transform, Global: link.Global, Local: link.Local})
			} else {
				mainInputs[link.Global] = link.Global
			}
		}

View on GitHub (pinned to 12126d8942)

Solutions

1. Ensure SDK and prism runner are the same Beam version
2. Check that the transform URN actually matches the payload type actually stored
3. Re-generate/re-submit the pipeline rather than reusing a cached/corrupted job submission

Example fix

// verify payload before decode
if len(t.GetSpec().GetPayload()) == 0 {
    return fmt.Errorf("empty ParDoPayload for %v", link.Transform)
}

if len(t.GetSpec().GetPayload()) == 0 {
    return fmt.Errorf("missing payload for ParDo %v", link.Transform)
}

if err := finalizeStage(&stg, comps); err != nil {
    slog.Error("stage finalize failed", "stage", stg.ID, "err", err)
    return err
}

When it happens

Trigger: A transform with a ParDo-family URN (TransformParDo, ProcessSizedElements, PairWithRestriction, SplitAndSize, Truncate) whose GetSpec().GetPayload() bytes are not a valid ParDoPayload proto.

Common situations: Beam SDK/runner version skew where payload encoding changed; pipeline constructed by another tool that attached wrong payloads; corrupted serialized pipelines.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• retrieveCoders: couldn't handle component %d %q of %q %v: %w
• prism error building stage %v - decoding TestStreamPayload:
• unable to unmarshal ParDoPayload for %v - %q: %w
• computeFacts: unable to check %q side inputs
• unable to decode ParDoPayload

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/34a690669d6ed544. Report an issue: GitHub.