apache/beam · error

unable to decode ParDoPayload for transform

Error message

unable to decode ParDoPayload for transform[%v]

What it means

The prism runner's PrepareTransform tries to unmarshal the transform's spec payload into a pipepb.ParDoPayload before handling the ParDo. If the payload bytes cannot be decoded as a ParDoPayload protobuf, the runner panics rather than continuing, because the rest of ParDo handling depends on fields like RestrictionCoderId.

Solutions

1. Verify the SDK and prism runner versions come from the same Beam release so the ParDoPayload proto matches.
2. Inspect the failing transform's URN and spec payload (prototext/protoc) to confirm it is a well-formed ParDoPayload.
3. Rebuild/regenerate the pipeline so the ParDo spec payload is populated by the SDK's standard transform-creation path, not hand-constructed.
4. If it persists, file a Beam issue with the pipeline graph dump; this is normally an SDK-side serialization bug.

if t.GetSpec().GetUrn() == urns.TransformParDo && len(t.GetSpec().GetPayload()) == 0 {
    // skip/submit error before invoking prism
}
pdo := &pipepb.ParDoPayload{}
if err := proto.Unmarshal(t.GetSpec().GetPayload(), pdo); err != nil { /* handle */ }

func isDecodableParDo(t *pipepb.PTransform) bool { p := t.GetSpec().GetPayload(); return p != nil && t.GetSpec().GetUrn() == urns.TransformParDo }

defer func(){ if r := recover(); r != nil && strings.Contains(fmt.Sprint(r), "unable to decode ParDoPayload") { /* handle submit failure */ } }()

When it happens

Trigger: A pipeline is submitted to the prism runner with a transform whose URN is a ParDo URN but whose spec payload is empty, truncated, corrupt, or was serialized by an SDK producing an incompatible/older ParDoPayload encoding.

Common situations: Running a pipeline with an SDK/runner version mismatch where the protobuf definitions diverged; a custom or hand-crafted pipeline graph (e.g. from a test harness or job-submission tool) that sets a wrong payload for a ParDo transform.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• computeFacts: unable to check
• empty type
• Failed to decode TestStreamPayload:
• failed to decode userfn
• failed to marshal payload as proto

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/92cf223a610c972d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/prism/internal/handlepardo.go:77

// PrepareTransform handles special processing with respect to ParDos, since their handling is dependant on supported features
// and requirements.
func (h *pardo) PrepareTransform(tid string, t *pipepb.PTransform, comps *pipepb.Components) prepareResult {

	// ParDos are a pain in the butt.
	// Combines, by comparison, are dramatically simpler.
	// This is because for ParDos, how they are handled, and what kinds of transforms are in
	// and around the ParDo, the actual shape of the graph will change.
	// At their simplest, it's something a DoFn will handle on their own.
	// At their most complex, they require intimate interaction with the subgraph
	// bundling process, the data layer, state layers, and control layers.
	// But unlike combines, which have a clear urn for composite + special payload,
	// ParDos have the standard URN for composites with the standard payload.
	// So always, we need to first unmarshal the payload.

	pardoPayload := t.GetSpec().GetPayload()
	pdo := &pipepb.ParDoPayload{}
	if err := (proto.UnmarshalOptions{}).Unmarshal(pardoPayload, pdo); err != nil {
		panic(fmt.Sprintf("unable to decode ParDoPayload for transform[%v]", t.GetUniqueName()))
	}

	// Lets check for and remove anything that makes things less simple.
	if pdo.RestrictionCoderId == "" {
		// Which inputs are Side inputs don't change the graph further,
		// so they're not included here. Any nearly any ParDo can have them.

		// At their simplest, we don't need to do anything special at pre-processing time, and simply pass through as normal.

		// ForceRoots cause fusion breaks in the optimized graph.
		// StatefulDoFns need to be marked as being roots, for correct per-key state handling.
		// Prism already sorts input elements for a stage by EventTime, so a fusion break enables the sorted behavior.
		var forcedRoots []string
		if len(pdo.GetStateSpecs())+len(pdo.GetTimerFamilySpecs()) > 0 ||
			pdo.GetRequiresTimeSortedInput() {
			forcedRoots = append(forcedRoots, tid)
		}

View on GitHub (pinned to 12126d8942)