apache/beam · critical

failed to create GCS client

Error message

failed to create GCS client

What it means

gcs.New creates a GCS filesystem using an authenticated storage client; if that fails it warns and falls back to an unauthenticated client, and if that also fails it panics with this wrapped error because no GCS access is possible at all.

Solutions

1. Set up Application Default Credentials: gcloud auth application-default login, or point GOOGLE_APPLICATION_CREDENTIALS at a valid service-account JSON key
2. Ensure the environment has OAuth endpoints reachable and, on GCP, the correct access scopes (storage read/write, cloud-platform)
3. Verify the credentials JSON is valid: the key file exists, is parseable, and the service account is enabled
4. If anonymous access is intended, confirm the bucket is publicly readable; note NewUnauthenticatedClient cannot access private buckets
5. Check the preceding log line 'falling back to unauthenticated GCS access' for the root authenticated-client failure and fix that

Example fix

// before (environment)
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/missing.json
// after
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/valid-service-account.json
# or locally:
gcloud auth application-default login

creds := os.Getenv("GOOGLE_APPLICATION_CREDENTIALS")
if creds == "" {
    if _, err := os.Stat(filepath.Join(home(), ".config/gcloud/application_default_credentials.json")); err != nil {
        return errors.New("no ADC found; run 'gcloud auth application-default login'")
    }
} else if _, err := os.Stat(creds); err != nil {
    return fmt.Errorf("credential file missing: %s", creds)
}
if _, err := tokenugar.NewFileTokenSource(creds); err != nil { return err }

// gcs.New panics; recover at job setup
func newFS(ctx context.Context) (fs Interface) {
    defer func() {
        if r := recover(); r != nil {
            log.Fatalf("GCS filesystem init failed: %v", r)
        }
    }()
    return gcs.New(ctx)
}

When it happens

Trigger: Both gcsx.NewClient(ctx, storage.ScopeReadWrite) and gcsx.NewUnauthenticatedClient(ctx) return errors: no usable token source (broken ADC, missing metadata server on non-GCP environments, malformed GOOGLE_APPLICATION_CREDENTIALS) for the first, and construction of the anonymous client failing for the second (rare; typically only on unsupported platforms/blocked endpoints).

Common situations: Running a Beam pipeline locally without 'gcloud auth application-default login'; GOOGLE_APPLICATION_CREDENTIALS pointing at a missing/invalid JSON key; Dataflow worker service account lacking cloud-platform scopes; metadata server unreachable in containers without GCE metadata emulation.

Related errors

• bucket must be non-empty
• could not resolve to a temp directory for import batch files
• Could not upload to GCS path
• Could not upload to GCS path
• empty chunk

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/48d30ab2932bdf83. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/filesystem/gcs/gcs.go:151

	hooks.RegisterHook(projectBillingHook, hf)
}

type fs struct {
	client *storage.Client
}

// New creates a new Google Cloud Storage filesystem using application
// default credentials. If it fails, it falls back to unauthenticated
// access.
// It will use the environment variable named `BILLING_PROJECT_ID` as requester payer bucket attribute.
func New(ctx context.Context) filesystem.Interface {
	client, err := gcsx.NewClient(ctx, storage.ScopeReadWrite)
	if err != nil {
		log.Warnf(ctx, "Warning: falling back to unauthenticated GCS access: %v", err)

		client, err = gcsx.NewUnauthenticatedClient(ctx)
		if err != nil {
			panic(errors.Wrapf(err, "failed to create GCS client"))
		}
	}
	return &fs{
		client: client,
	}
}

func SetRequesterBillingProject(project string) {
	billingProject = project
}

// RequesterBillingProject configure project to be used in google storage operations
// with requester pays actived. More informaiton about requester pays in https://cloud.google.com/storage/docs/requester-pays
func RequesterBillingProject(project string) error {
	if project == "" {
		return fmt.Errorf("project cannot be empty, got %v", project)
	}
	// The hook itself is defined in beam/core/runtime/harness/file_system_hooks.go

View on GitHub (pinned to 12126d8942)