{"record":{"id":"699127c0660edd10","repo":"apache/beam","slug":"failed-to-write-manifest","errorCode":null,"errorMessage":"failed to write manifest","messagePattern":"failed to write manifest","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/artifact/gcsproxy/staging.go","lineNumber":88,"sourceCode":"\ts.mu.Lock()\n\tloc, err := matchLocations(manifest.GetArtifact(), s.blobs)\n\tif err != nil {\n\t\ts.mu.Unlock()\n\t\treturn nil, err\n\t}\n\ts.mu.Unlock()\n\n\tdata, err := proto.Marshal(&jobpb.ProxyManifest{Manifest: manifest, Location: loc})\n\tif err != nil {\n\t\treturn nil, errors.Wrap(err, \"failed to marshal proxy manifest\")\n\t}\n\n\tcl, err := gcsx.NewClient(ctx, storage.ScopeReadWrite)\n\tif err != nil {\n\t\treturn nil, errors.Wrap(err, \"failed to create GCS client\")\n\t}\n\tif err := gcsx.WriteObject(ctx, cl, s.bucket, s.manifest, bytes.NewReader(data)); err != nil {\n\t\treturn nil, errors.Wrap(err, \"failed to write manifest\")\n\t}\n\n\t// Commit returns the location of the manifest as the token, which can\n\t// then be used to configure the retrieval proxy. It is redundant right\n\t// now, but would be needed for a staging server that serves multiple\n\t// jobs. Such a server would also use the ID sent with each request.\n\n\treturn &jobpb.CommitManifestResponse{RetrievalToken: gcsx.MakeObject(s.bucket, s.manifest)}, nil\n}\n\n// matchLocations ensures that all artifacts have been staged and have valid\n// content. It is fine for staged artifacts to not appear in the manifest.\nfunc matchLocations(artifacts []*jobpb.ArtifactMetadata, blobs map[string]staged) ([]*jobpb.ProxyManifest_Location, error) {\n\tvar loc []*jobpb.ProxyManifest_Location\n\tfor _, a := range artifacts {\n\t\tinfo, ok := blobs[a.Name]\n\t\tif !ok {\n\t\t\treturn nil, errors.Errorf(\"artifact %v not staged\", a.Name)","sourceCodeStart":70,"sourceCodeEnd":106,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/artifact/gcsproxy/staging.go#L70-L106","documentation":"CommitManifest writes the marshaled ProxyManifest object to the bucket/object derived from the manifest path via gcsx.WriteObject. Any upload failure (permissions, bucket missing, transient GCS errors) is wrapped with this message; staging cannot complete without committing the manifest.","triggerScenarios":"gcsx.WriteObject(ctx, cl, s.bucket, s.manifest, bytes.NewReader(data)) returns non-nil during CommitManifest, after a successful GCS client creation and artifact upload.","commonSituations":"Service account lacks storage.objects.create on the bucket; bucket was deleted or renamed after staging began; bucket name typo'd in --staging_location; transient GCS 5xx/429; CMEK/retention policies blocking writes; bucket region mismatch with request.","solutions":["Grant the staging identity storage.objects.create/permissions (roles/storage.objectAdmin or objectCreator) on the bucket.","Verify the bucket exists and the spelling of the staging location (gsutil ls gs://bucket).","Retry on transient GCS errors (5xx, rate limits).","Check bucket policies (retention, CMEK, uniform access) that could reject the write.","Confirm network/firewall allows uploads to storage.googleapis.com."],"exampleFix":"// before\nerr := gcsx.WriteObject(ctx, cl, s.bucket, s.manifest, bytes.NewReader(data))\nif err != nil {\n\treturn nil, errors.Wrap(err, \"failed to write manifest\")\n}\n// after\nvar gerr error\nfor i := 0; i < 3; i++ {\n\tgerr = gcsx.WriteObject(ctx, cl, s.bucket, s.manifest, bytes.NewReader(data))\n\tif gerr == nil {\n\t\tbreak\n\t}\n\ttime.Sleep(time.Duration(1<<i) * time.Second)\n}\nif gerr != nil {\n\treturn nil, errors.Wrap(gerr, \"failed to write manifest\")\n}","handlingStrategy":"retry","validationCode":"// Check write permission before staging\nit := gcsClient.Bucket(bucket).Objects(ctx, nil)\n_, err := it.Next() // err==nil or iterator.Done means bucket exists & is listable","typeGuard":null,"tryCatchPattern":"if err := commit(...); err != nil {\n\tif strings.Contains(err.Error(), \"failed to write manifest\") {\n\t\t// check bucket existence/permissions; retry with backoff on 5xx/429\n\t}\n}","preventionTips":["Grant the staging identity objectCreator (or higher) on the staging bucket.","Verify the bucket exists before launching staged pipelines.","Retry transient GCS write errors with exponential backoff.","Keep staging buckets free of restrictive retention/CMEK policies for pipeline use."],"tags":["gcs","storage","permissions","upload"],"backgroundTag":"file-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}