apache/beam · error

failed to create GCS client

Error message

failed to create GCS client

What it means

ReadProxyManifest creates a read-only Google Cloud Storage client after parsing the object path. This error wraps the underlying client-creation failure, meaning authentication or client construction failed before any manifest was read.

Solutions

1. Set up Application Default Credentials: run gcloud auth application-default login locally or attach a service account with Storage Object Viewer.
2. Set GOOGLE_APPLICATION_CREDENTIALS to a valid service-account JSON key file.
3. Confirm the Cloud Storage API is enabled in the project and the environment has network access to storage.googleapis.com.

Example fix

// before (no credentials configured)
md, err := ReadProxyManifest(ctx, objPath)
// after
os.Setenv("GOOGLE_APPLICATION_CREDENTIALS", "/path/to/sa.json")
md, err := ReadProxyManifest(ctx, objPath)

creds := os.Getenv("GOOGLE_APPLICATION_CREDENTIALS")
if creds == "" { if _, err := gcp.DefaultCredentials(ctx); err != nil { return fmt.Errorf("no GCS credentials: %w", err) } }

var md *jobpb.ProxyManifest
err := retry.Exponential(ctx, 3, func() error { var e error; md, e = ReadProxyManifest(ctx, obj); return e })

When it happens

Trigger: Calling ReadProxyManifest when gcsx.NewClient fails: missing Application Default Credentials, GOOGLE_APPLICATION_CREDENTIALS unset/invalid, or the Cloud Storage client library failing to initialize.

Common situations: Running outside GCE/Cloud Run without a service account, expired or malformed credential JSON files, or disabled Cloud Storage API on the project.

Related errors

• Failed to create client for
• failed to read manifest
• Failed to read object for
• Artifact not found at
• Can not get unique key from solr

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ade3d111f7170728. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go:46

// RetrievalServer is a artifact retrieval server backed by Google
// Cloud Storage (GCS). It serves a single manifest and ignores
// the worker id. The server performs no caching or pre-fetching.
type RetrievalServer struct {
	md    *jobpb.Manifest
	blobs map[string]string
}

// ReadProxyManifest reads and parses the proxy manifest from GCS.
func ReadProxyManifest(ctx context.Context, object string) (*jobpb.ProxyManifest, error) {
	bucket, obj, err := gcsx.ParseObject(object)
	if err != nil {
		return nil, errors.Wrapf(err, "invalid manifest object %v", object)
	}

	cl, err := gcsx.NewClient(ctx, storage.ScopeReadOnly)
	if err != nil {
		return nil, errors.Wrap(err, "failed to create GCS client")
	}
	content, err := gcsx.ReadObject(ctx, cl, bucket, obj)
	if err != nil {
		return nil, errors.Wrapf(err, "failed to read manifest %v", object)
	}
	var md jobpb.ProxyManifest
	if err := proto.Unmarshal(content, &md); err != nil {
		return nil, errors.Wrapf(err, "invalid manifest %v", object)
	}
	return &md, nil
}

// NewRetrievalServer creates a artifact retrieval server for the
// given manifest. It requires that the locations are in GCS.
func NewRetrievalServer(md *jobpb.ProxyManifest) (*RetrievalServer, error) {
	if err := validate(md); err != nil {
		return nil, err
	}

View on GitHub (pinned to 12126d8942)