apache/beam · error

failed to marshal proxy manifest

Error message

failed to marshal proxy manifest

What it means

CommitManifest serializes the ProxyManifest (artifacts plus their GCS locations) with proto.Marshal before uploading it to GCS. This error wraps a proto marshaling failure, which is rare but can occur if the message tree contains invalid data that cannot be encoded.

Solutions

1. Check that artifact and location entries added during staging contain valid, non-nil fields.
2. Ensure the generated jobpb/pipepb packages come from the same Beam version to avoid proto descriptor mismatches.
3. Log manifest.Artifact and loc contents before marshaling to spot invalid entries.
4. Retry the pipeline; if reproducible, file an issue with the artifact names/URIs involved.

// Pre-check manifest entries are sane before commit
for _, a := range manifest.GetArtifact() {
	if a.Name == "" { return errors.New("artifact with empty name") }
}

data, err := proto.Marshal(md)
if err != nil {
	return nil, fmt.Errorf("proxy manifest marshal failed: %w", err)
}

When it happens

Trigger: proto.Marshal(&jobpb.ProxyManifest{Manifest: manifest, Location: loc}) returns non-nil during CommitManifest, after all staged artifacts have been recorded.

Common situations: Corrupt or nil-required fields introduced into the manifest by custom staging code; incompatible generated proto types from mismatched beam versions in a custom build; memory corruption is effectively unheard of — usually a custom-modified message.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• Could not encode message as bytes
• empty type
• Failed to convert PipelineOptions to JSON
• failed to create artifact role payload
• failed to marshal payload as proto

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/fef35d1694b65293. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/artifact/gcsproxy/staging.go:80

		blobs:    make(map[string]staged),
	}, nil
}

// CommitManifest commits the given artifact manifest to GCS.
func (s *StagingServer) CommitManifest(ctx context.Context, req *jobpb.CommitManifestRequest) (*jobpb.CommitManifestResponse, error) {
	manifest := req.GetManifest()

	s.mu.Lock()
	loc, err := matchLocations(manifest.GetArtifact(), s.blobs)
	if err != nil {
		s.mu.Unlock()
		return nil, err
	}
	s.mu.Unlock()

	data, err := proto.Marshal(&jobpb.ProxyManifest{Manifest: manifest, Location: loc})
	if err != nil {
		return nil, errors.Wrap(err, "failed to marshal proxy manifest")
	}

	cl, err := gcsx.NewClient(ctx, storage.ScopeReadWrite)
	if err != nil {
		return nil, errors.Wrap(err, "failed to create GCS client")
	}
	if err := gcsx.WriteObject(ctx, cl, s.bucket, s.manifest, bytes.NewReader(data)); err != nil {
		return nil, errors.Wrap(err, "failed to write manifest")
	}

	// Commit returns the location of the manifest as the token, which can
	// then be used to configure the retrieval proxy. It is redundant right
	// now, but would be needed for a staging server that serves multiple
	// jobs. Such a server would also use the ID sent with each request.

	return &jobpb.CommitManifestResponse{RetrievalToken: gcsx.MakeObject(s.bucket, s.manifest)}, nil
}

View on GitHub (pinned to 12126d8942)