apache/beam · error

failed to marshal payload as proto

Error message

failed to marshal payload as proto

What it means

EncodeStructPayload serializes an ExpansionRequest/external payload struct to protobuf bytes for cross-language transform payloads. This error is returned when proto.Marshal fails on the constructed payload proto, wrapped with context naming the payload. proto.Marshal on a generated struct rarely fails, so this usually indicates a corrupted or unsupported message (e.g. containing an unmarshalable Any or invalid proto state).

Source

Thrown at sdks/go/pkg/beam/core/runtime/xlangx/payload.go:74

		Schema:  scm,
		Payload: buf.Bytes(),
	}
	return ecp, nil
}

// EncodeStructPayload takes a native Go struct and returns a marshaled
// ExternalConfigurationPayload proto, containing a Schema representation of
// the original type and the original value encoded as a Row. This is intended
// to be used as the expansion payload for an External transform.
func EncodeStructPayload(pl any) ([]byte, error) {
	ecp, err := CreateExternalConfigurationPayload(pl)
	if err != nil {
		return []byte{}, err
	}

	plBytes, err := proto.Marshal(ecp)
	if err != nil {
		err = errors.Wrapf(err, "failed to marshal payload as proto")
		return []byte{}, errors.WithContextf(err, "encoding external payload %v", pl)
	}

	return plBytes, nil
}

// DecodeStructPayload takes a marshaled ExternalConfigurationPayload proto
// and returns a native Go struct, with its type converted from the Schema
// representation and its value decoded from the Row.
func DecodeStructPayload(plBytes []byte) (any, error) {
	// Unmarshal payload proto.
	ecp := &pipepb.ExternalConfigurationPayload{}
	if err := proto.Unmarshal(plBytes, ecp); err != nil {
		err = errors.WithContext(err, "failed to unmarshal the payload proto")
		return nil, errors.WithContext(err, "decoding external payload")
	}

	// Convert Schema representation into payload type.

View on GitHub (pinned to 12126d8942)

Solutions

1. Check the wrapped cause (errors.Unwrap / %v of the returned error) to see why Marshal failed and fix the offending field.
2. Ensure the payload struct is a protobuf-generated message compatible with the google.golang.org/protobuf runtime in use.
3. Pin/upgrade google.golang.org/protobuf and github.com/golang/protobuf to consistent versions.
4. If constructing payloads by hand, validate fields (non-nil oneof case, registered Any types) before calling EncodeStructPayload.

Example fix

// before
payload := myCustomGoStruct{Data: data}
b, err := xlangx.EncodeStructPayload(payload)

// after
payload := &pipepb.ExpansionRequest{Components: components, Namespace: ns}
b, err := xlangx.EncodeStructPayload(payload)

if _, ok := payload.(proto.Message); !ok {
    return fmt.Errorf("payload %T is not a proto.Message", payload)
}
if _, err := proto.Marshal(payload.(proto.Message)); err != nil {
    return err // fail before the real call
}

func isProtoMessage(v any) bool { _, ok := v.(proto.Message); return ok }

b, err := xlangx.EncodeStructPayload(payload)
if err != nil {
    var cause error
    for cause = err; errors.Unwrap(cause) != nil; cause = errors.Unwrap(cause) {
    }
    return fmt.Errorf("payload encoding failed for %T: %w", payload, cause)
}

When it happens

Trigger: Calling xlangx.EncodeStructPayload / CrossLanguagePayload with a struct that cannot be marshaled by the protobuf runtime — e.g. a message containing an invalid google.protobuf.Any, unregistered dynamic message, or nil in a required oneof state.

Common situations: Passing a wrong type or a struct not generated by protoc-gen-go as the payload; mixing incompatible google.golang.org/protobuf versions (old github.com/golang/protobuf vs new APIv2); payloads built programmatically with reflection.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• empty type
• failed to create artifact role payload
• invalid float encoding for: %v
• invalid ParDo payload for %v
• invalid CombinePayload payload for %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d9da920e1e0a9030. Report an issue: GitHub.