apache/beam · error

failed to write manifest

Error message

failed to write manifest

What it means

CommitManifest writes the marshaled ProxyManifest object to the bucket/object derived from the manifest path via gcsx.WriteObject. Any upload failure (permissions, bucket missing, transient GCS errors) is wrapped with this message; staging cannot complete without committing the manifest.

Solutions

1. Grant the staging identity storage.objects.create/permissions (roles/storage.objectAdmin or objectCreator) on the bucket.
2. Verify the bucket exists and the spelling of the staging location (gsutil ls gs://bucket).
3. Retry on transient GCS errors (5xx, rate limits).
4. Check bucket policies (retention, CMEK, uniform access) that could reject the write.
5. Confirm network/firewall allows uploads to storage.googleapis.com.

Example fix

// before
err := gcsx.WriteObject(ctx, cl, s.bucket, s.manifest, bytes.NewReader(data))
if err != nil {
	return nil, errors.Wrap(err, "failed to write manifest")
}
// after
var gerr error
for i := 0; i < 3; i++ {
	gerr = gcsx.WriteObject(ctx, cl, s.bucket, s.manifest, bytes.NewReader(data))
	if gerr == nil {
		break
	}
	time.Sleep(time.Duration(1<<i) * time.Second)
}
if gerr != nil {
	return nil, errors.Wrap(gerr, "failed to write manifest")
}

// Check write permission before staging
it := gcsClient.Bucket(bucket).Objects(ctx, nil)
_, err := it.Next() // err==nil or iterator.Done means bucket exists & is listable

if err := commit(...); err != nil {
	if strings.Contains(err.Error(), "failed to write manifest") {
		// check bucket existence/permissions; retry with backoff on 5xx/429
	}
}

When it happens

Trigger: gcsx.WriteObject(ctx, cl, s.bucket, s.manifest, bytes.NewReader(data)) returns non-nil during CommitManifest, after a successful GCS client creation and artifact upload.

Common situations: Service account lacks storage.objects.create on the bucket; bucket was deleted or renamed after staging began; bucket name typo'd in --staging_location; transient GCS 5xx/429; CMEK/retention policies blocking writes; bucket region mismatch with request.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Could not upload to GCS path
• Could not upload to GCS path
• failed to read from
• Unable create default bucket.
• Unable to determine the owner of the default bucket at gs://

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/699127c0660edd10. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/artifact/gcsproxy/staging.go:88

	s.mu.Lock()
	loc, err := matchLocations(manifest.GetArtifact(), s.blobs)
	if err != nil {
		s.mu.Unlock()
		return nil, err
	}
	s.mu.Unlock()

	data, err := proto.Marshal(&jobpb.ProxyManifest{Manifest: manifest, Location: loc})
	if err != nil {
		return nil, errors.Wrap(err, "failed to marshal proxy manifest")
	}

	cl, err := gcsx.NewClient(ctx, storage.ScopeReadWrite)
	if err != nil {
		return nil, errors.Wrap(err, "failed to create GCS client")
	}
	if err := gcsx.WriteObject(ctx, cl, s.bucket, s.manifest, bytes.NewReader(data)); err != nil {
		return nil, errors.Wrap(err, "failed to write manifest")
	}

	// Commit returns the location of the manifest as the token, which can
	// then be used to configure the retrieval proxy. It is redundant right
	// now, but would be needed for a staging server that serves multiple
	// jobs. Such a server would also use the ID sent with each request.

	return &jobpb.CommitManifestResponse{RetrievalToken: gcsx.MakeObject(s.bucket, s.manifest)}, nil
}

// matchLocations ensures that all artifacts have been staged and have valid
// content. It is fine for staged artifacts to not appear in the manifest.
func matchLocations(artifacts []*jobpb.ArtifactMetadata, blobs map[string]staged) ([]*jobpb.ProxyManifest_Location, error) {
	var loc []*jobpb.ProxyManifest_Location
	for _, a := range artifacts {
		info, ok := blobs[a.Name]
		if !ok {
			return nil, errors.Errorf("artifact %v not staged", a.Name)

View on GitHub (pinned to 12126d8942)