apache/beam · error

unable to generate proto representation of %v

Error message

unable to generate proto representation of %v

What it means

Raised in xlangx.Expand when graphx.Marshal fails to serialize the MultiEdge for a cross-language transform into its protobuf (ExpansionRequest) representation. The library wraps the underlying marshal error with the external transform for context. It means the pipeline graph could not be converted to proto before being sent to an expansion service.

Source

Thrown at sdks/go/pkg/beam/core/runtime/xlangx/expand.go:56

// maxRetries is the maximum number of retries to attempt connecting to
// an expansion service endpoint.
const maxRetries = 5

// Expand expands an unexpanded graph.ExternalTransform as a
// graph.ExpandedTransform and assigns it to the ExternalTransform's Expanded
// field. This requires querying an expansion service based on the configuration
// details within the ExternalTransform.
//
// For framework use only. Users should call beam.CrossLanguage to access foreign transforms
// rather than calling this function directly.
func Expand(edge *graph.MultiEdge, ext *graph.ExternalTransform) error {
	// Build the ExpansionRequest

	// Obtaining the components and transform proto representing this transform
	p, err := graphx.Marshal([]*graph.MultiEdge{edge}, &graphx.Options{})
	if err != nil {
		return errors.Wrapf(err, "unable to generate proto representation of %v", ext)
	}

	transforms := p.GetComponents().GetTransforms()

	// Transforms consist of only External transform and composites. Composites
	// should be removed from proto before submitting expansion request.
	extTransformID := p.GetRootTransformIds()[0]
	extTransform := transforms[extTransformID]
	for extTransform.UniqueName != "External" {
		delete(transforms, extTransformID)
		p, err = pipelinex.Normalize(p) // Update root transform IDs.
		if err != nil {
			return err
		}
		transforms = p.GetComponents().GetTransforms()
		extTransformID = p.GetRootTransformIds()[0]
		extTransform = transforms[extTransformID]
	}

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the wrapped cause (errors.Wrapf preserves it) to see which component failed to marshal
2. Check that all PCollections use standard Beam coders supported by the Go SDK proto representation
3. Upgrade to a recent Beam Go SDK version where graphx supports more transform/coder types
4. Simplify the transform's inputs/outputs (avoid exotic composite edges) before cross-language expansion

Example fix

// before
beam.CrossLanguage(s, "pkg:Transform", cfg, beam.WindowingStrategyDefault, in, customCodedOut) // custom coder unsupported
// after
out := beam.CrossLanguage(s, "pkg:Transform", cfg, beam.WindowingStrategyDefault, in) // use standard coders on the boundary

if edge == nil || ext == nil {
    return fmt.Errorf("cross-language edge/transform not initialized")
}

if err := xlangx.Expand(edge, ext); err != nil {
    var cause error
    errors.As(err, &cause) // inspect wrapped marshal cause
    return fmt.Errorf("cross-language expansion setup failed: %w", err)
}

When it happens

Trigger: Calling beam.CrossLanguage (which reaches TryCrossLanguage -> Expand) with a transform whose graph elements cannot be marshaled by graphx.Marshal, e.g. a MultiEdge containing unsupported input/output types or an unencodable coder.

Common situations: Using a cross-language wrapper with custom coders or PCollection types not supported by the Go proto marshaller; pipeline graphs built with constructs the marshaller does not recognize; version skew where graphx cannot represent newer edge features.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• empty type
• failed to create artifact role payload
• unexpected type kind %v
• failed to marshal payload as proto
• Could not encode message as bytes

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/60400a5a8766a77d. Report an issue: GitHub.