apache/beam · error

failed to read manifest

Error message

failed to read manifest %v

What it means

After creating the GCS client, ReadProxyManifest downloads the manifest object. This wrapped error is returned when gcsx.ReadObject fails to fetch the object bytes, e.g. the object does not exist or is not readable.

Solutions

1. Verify the object exists: gsutil ls gs://bucket/path/to/manifest.
2. Grant the caller's service account storage.objects.get (roles/storage.objectViewer) on the bucket.
3. Check the path casing/spelling and that the job that wrote the manifest completed successfully.

Example fix

// before (object not yet uploaded)
md, _ := ReadProxyManifest(ctx, manifestPath)
// after: ensure writer finished before reading
if err := waitForManifest(manifestPath); err != nil { return err }
md, err := ReadProxyManifest(ctx, manifestPath)

func manifestExists(ctx context.Context, obj string) bool { cl, err := gcsx.NewClient(ctx, storage.ScopeReadOnly); if err != nil { return false }; b, o, err := gcsx.ParseObject(obj); if err != nil { return false }; _, err = cl.Bucket(b).Object(o).Attrs(ctx); return err == nil }

md, err := ReadProxyManifest(ctx, obj)
if err != nil && strings.Contains(err.Error(), "failed to read manifest") { return fmt.Errorf("manifest %s missing/unreadable: %w", obj, err) }

When it happens

Trigger: Calling ReadProxyManifest with a gs:// path whose object is missing, deleted, or inaccessible to the credentials (403/404 from GCS during read).

Common situations: Typo in bucket or object name, manifest written by a different job run was garbage-collected, or the service account lacks storage.objects.get on the object.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Failed to read object for
• Failed to create client for
• failed to create GCS client
• Artifact not found at
• Can not get unique key from solr

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/295f1d5b199a7de7. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go:50

type RetrievalServer struct {
	md    *jobpb.Manifest
	blobs map[string]string
}

// ReadProxyManifest reads and parses the proxy manifest from GCS.
func ReadProxyManifest(ctx context.Context, object string) (*jobpb.ProxyManifest, error) {
	bucket, obj, err := gcsx.ParseObject(object)
	if err != nil {
		return nil, errors.Wrapf(err, "invalid manifest object %v", object)
	}

	cl, err := gcsx.NewClient(ctx, storage.ScopeReadOnly)
	if err != nil {
		return nil, errors.Wrap(err, "failed to create GCS client")
	}
	content, err := gcsx.ReadObject(ctx, cl, bucket, obj)
	if err != nil {
		return nil, errors.Wrapf(err, "failed to read manifest %v", object)
	}
	var md jobpb.ProxyManifest
	if err := proto.Unmarshal(content, &md); err != nil {
		return nil, errors.Wrapf(err, "invalid manifest %v", object)
	}
	return &md, nil
}

// NewRetrievalServer creates a artifact retrieval server for the
// given manifest. It requires that the locations are in GCS.
func NewRetrievalServer(md *jobpb.ProxyManifest) (*RetrievalServer, error) {
	if err := validate(md); err != nil {
		return nil, err
	}

	blobs := make(map[string]string)
	for _, l := range md.GetLocation() {
		if _, _, err := gcsx.ParseObject(l.GetUri()); err != nil {

View on GitHub (pinned to 12126d8942)