apache/beam · error

invalid manifest

Error message

invalid manifest %v

What it means

ReadProxyManifest unmarshals the downloaded bytes as a protobuf ProxyManifest. This wrapped error is returned when proto.Unmarshal fails, meaning the object content is not a valid encoded ProxyManifest protobuf.

Solutions

1. Ensure the manifest was written as a serialized jobpb.ProxyManifest protobuf (proto.Marshal output), not JSON or YAML.
2. Re-generate/re-upload the manifest and retry.
3. Check that the writer and reader use compatible Beam versions for the ProxyManifest schema.

Example fix

// before: writing JSON manifest
json.NewEncoder(w).Encode(manifest)
// after: write protobuf bytes
b, _ := proto.Marshal(manifest)
w.Write(b)

if len(content) == 0 { return errors.New("manifest object is empty") }

md, err := ReadProxyManifest(ctx, obj)
if err != nil && strings.Contains(err.Error(), "invalid manifest") { return fmt.Errorf("manifest is not a valid ProxyManifest protobuf: %w", err) }

When it happens

Trigger: Calling ReadProxyManifest against an object whose bytes are not the serialized ProxyManifest (e.g. JSON manifest, empty object, truncated upload, or a text file).

Common situations: A pipeline wrote the manifest in a different format/version, an upload was interrupted leaving corrupt bytes, or a user manually placed a file at the manifest path.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• Could not parse Pub/Sub message
• bad function
• bad type: , want
• bad window
• base64 decode for failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4ebc127f720f2388. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go:54

// ReadProxyManifest reads and parses the proxy manifest from GCS.
func ReadProxyManifest(ctx context.Context, object string) (*jobpb.ProxyManifest, error) {
	bucket, obj, err := gcsx.ParseObject(object)
	if err != nil {
		return nil, errors.Wrapf(err, "invalid manifest object %v", object)
	}

	cl, err := gcsx.NewClient(ctx, storage.ScopeReadOnly)
	if err != nil {
		return nil, errors.Wrap(err, "failed to create GCS client")
	}
	content, err := gcsx.ReadObject(ctx, cl, bucket, obj)
	if err != nil {
		return nil, errors.Wrapf(err, "failed to read manifest %v", object)
	}
	var md jobpb.ProxyManifest
	if err := proto.Unmarshal(content, &md); err != nil {
		return nil, errors.Wrapf(err, "invalid manifest %v", object)
	}
	return &md, nil
}

// NewRetrievalServer creates a artifact retrieval server for the
// given manifest. It requires that the locations are in GCS.
func NewRetrievalServer(md *jobpb.ProxyManifest) (*RetrievalServer, error) {
	if err := validate(md); err != nil {
		return nil, err
	}

	blobs := make(map[string]string)
	for _, l := range md.GetLocation() {
		if _, _, err := gcsx.ParseObject(l.GetUri()); err != nil {
			return nil, errors.Wrapf(err, "location %v is not a GCS object", l.GetUri())
		}
		blobs[l.GetName()] = l.GetUri()
	}

View on GitHub (pinned to 12126d8942)