apache/beam · error

unable to unmarshal ParDoPayload for %v - %q: %w

Error message

unable to unmarshal ParDoPayload for %v - %q: %w

What it means

During job preparation, prism validates transform specs. For a ParDo transform it attempts to proto.Unmarshal the transform's spec payload into pipepb.ParDoPayload. If unmarshaling fails (payload missing, truncated, or not a ParDoPayload), the job is marked Failed and Prepare returns this wrapped error including the transform ID, unique name, and cause.

Source

Thrown at sdks/go/pkg/beam/runners/prism/internal/jobservices/management.go:158

		case urns.TransformImpulse,
			urns.TransformGBK,
			urns.TransformFlatten,
			urns.TransformCombinePerKey,
			urns.TransformCombineGlobally,      // Used by Java SDK
			urns.TransformCombineGroupedValues, // Used by Java SDK
			urns.TransformMerge,                // Used directly by Python SDK if "pre-optimized"
			urns.TransformPreCombine,           // Used directly by Python SDK if "pre-optimized"
			urns.TransformExtract,              // Used directly by Python SDK if "pre-optimized"
			urns.TransformAssignWindows:
		// Very few expected transforms types for submitted pipelines.
		// Most URNs are for the runner to communicate back to the SDK for execution.
		case urns.TransformReshuffle, urns.TransformRedistributeArbitrarily, urns.TransformRedistributeByKey:
			// Reshuffles and Redistributes are permitted and have special handling during optimization.

		case urns.TransformParDo:
			var pardo pipepb.ParDoPayload
			if err := proto.Unmarshal(t.GetSpec().GetPayload(), &pardo); err != nil {
				wrapped := fmt.Errorf("unable to unmarshal ParDoPayload for %v - %q: %w", tid, t.GetUniqueName(), err)
				job.Failed(wrapped)
				return nil, wrapped
			}

			isStateful := false

			// Validate all the state features
			for _, spec := range pardo.GetStateSpecs() {
				isStateful = true
				check("StateSpec.Protocol.Urn", spec.GetProtocol().GetUrn(),
					urns.UserStateBag, urns.UserStateMultiMap, urns.UserStateOrderedList)
			}
			// Validate all the timer features
			for _, spec := range pardo.GetTimerFamilySpecs() {
				isStateful = true
				check("TimerFamilySpecs.TimeDomain.Urn", spec.GetTimeDomain(), pipepb.TimeDomain_EVENT_TIME, pipepb.TimeDomain_PROCESSING_TIME)
			}

View on GitHub (pinned to 12126d8942)

Solutions

1. Check the wrapped (%w) error for the concrete proto unmarshal cause.
2. Ensure the pipeline is generated and serialized by an SDK version compatible with the prism runner's Beam proto definitions.
3. Verify the transform's spec payload is actually a serialized ParDoPayload (not another payload type) and regenerate the pipeline.

Example fix

// before: wrong payload type on a ParDo URN
t.spec.payload = anypb(anotherPayload)

// after
t.spec.payload = pipepb.ParDoPayload{...} // correctly serialized ParDoPayload

// Client-side: verify ParDo spec payloads unmarshal before submission
var pardo pipepb.ParDoPayload
if err := proto.Unmarshal(t.GetSpec().GetPayload(), &pardo); err != nil {
    log.Fatalf("transform %q spec is not a valid ParDoPayload: %v", t.GetUniqueName(), err)
}

if err := beamx.Run(ctx, p); err != nil && strings.Contains(err.Error(), "unable to unmarshal ParDoPayload") {
    log.Fatalf("pipeline graph serialization is broken; regenerate the pipeline: %v", err)
}

When it happens

Trigger: A pipeline submitted to prism contains a transform with URN urn:beam:transform:pardo:v1 whose spec payload cannot be unmarshaled as ParDoPayload, encountered in management.go's Prepare validation switch.

Common situations: SDK/proto version mismatch where the payload was serialized with an incompatible schema; hand-built pipeline protos with the wrong spec bytes; corrupt serialization when the graph is generated by a non-Go SDK.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• retrieveCoders: couldn't handle component %d %q of %q %v: %w
• prism error building stage %v - decoding TestStreamPayload:
• computeFacts: unable to check %q side inputs
• unable to decode ParDoPayload for %v
• unable to decode ParDoPayload

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1eb99684db463552. Report an issue: GitHub.