{"record":{"id":"48d30ab2932bdf83","repo":"apache/beam","slug":"failed-to-create-gcs-client-48d30a","errorCode":null,"errorMessage":"failed to create GCS client","messagePattern":"failed to create GCS client","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"sdks/go/pkg/beam/io/filesystem/gcs/gcs.go","lineNumber":151,"sourceCode":"\thooks.RegisterHook(projectBillingHook, hf)\n}\n\ntype fs struct {\n\tclient *storage.Client\n}\n\n// New creates a new Google Cloud Storage filesystem using application\n// default credentials. If it fails, it falls back to unauthenticated\n// access.\n// It will use the environment variable named `BILLING_PROJECT_ID` as requester payer bucket attribute.\nfunc New(ctx context.Context) filesystem.Interface {\n\tclient, err := gcsx.NewClient(ctx, storage.ScopeReadWrite)\n\tif err != nil {\n\t\tlog.Warnf(ctx, \"Warning: falling back to unauthenticated GCS access: %v\", err)\n\n\t\tclient, err = gcsx.NewUnauthenticatedClient(ctx)\n\t\tif err != nil {\n\t\t\tpanic(errors.Wrapf(err, \"failed to create GCS client\"))\n\t\t}\n\t}\n\treturn &fs{\n\t\tclient: client,\n\t}\n}\n\nfunc SetRequesterBillingProject(project string) {\n\tbillingProject = project\n}\n\n// RequesterBillingProject configure project to be used in google storage operations\n// with requester pays actived. More informaiton about requester pays in https://cloud.google.com/storage/docs/requester-pays\nfunc RequesterBillingProject(project string) error {\n\tif project == \"\" {\n\t\treturn fmt.Errorf(\"project cannot be empty, got %v\", project)\n\t}\n\t// The hook itself is defined in beam/core/runtime/harness/file_system_hooks.go","sourceCodeStart":133,"sourceCodeEnd":169,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/io/filesystem/gcs/gcs.go#L133-L169","documentation":"gcs.New creates a GCS filesystem using an authenticated storage client; if that fails it warns and falls back to an unauthenticated client, and if that also fails it panics with this wrapped error because no GCS access is possible at all.","triggerScenarios":"Both gcsx.NewClient(ctx, storage.ScopeReadWrite) and gcsx.NewUnauthenticatedClient(ctx) return errors: no usable token source (broken ADC, missing metadata server on non-GCP environments, malformed GOOGLE_APPLICATION_CREDENTIALS) for the first, and construction of the anonymous client failing for the second (rare; typically only on unsupported platforms/blocked endpoints).","commonSituations":"Running a Beam pipeline locally without 'gcloud auth application-default login'; GOOGLE_APPLICATION_CREDENTIALS pointing at a missing/invalid JSON key; Dataflow worker service account lacking cloud-platform scopes; metadata server unreachable in containers without GCE metadata emulation.","solutions":["Set up Application Default Credentials: gcloud auth application-default login, or point GOOGLE_APPLICATION_CREDENTIALS at a valid service-account JSON key","Ensure the environment has OAuth endpoints reachable and, on GCP, the correct access scopes (storage read/write, cloud-platform)","Verify the credentials JSON is valid: the key file exists, is parseable, and the service account is enabled","If anonymous access is intended, confirm the bucket is publicly readable; note NewUnauthenticatedClient cannot access private buckets","Check the preceding log line 'falling back to unauthenticated GCS access' for the root authenticated-client failure and fix that"],"exampleFix":"// before (environment)\nexport GOOGLE_APPLICATION_CREDENTIALS=/path/to/missing.json\n// after\nexport GOOGLE_APPLICATION_CREDENTIALS=/path/to/valid-service-account.json\n# or locally:\ngcloud auth application-default login","handlingStrategy":"validation","validationCode":"creds := os.Getenv(\"GOOGLE_APPLICATION_CREDENTIALS\")\nif creds == \"\" {\n    if _, err := os.Stat(filepath.Join(home(), \".config/gcloud/application_default_credentials.json\")); err != nil {\n        return errors.New(\"no ADC found; run 'gcloud auth application-default login'\")\n    }\n} else if _, err := os.Stat(creds); err != nil {\n    return fmt.Errorf(\"credential file missing: %s\", creds)\n}\nif _, err := tokenugar.NewFileTokenSource(creds); err != nil { return err }","typeGuard":null,"tryCatchPattern":"// gcs.New panics; recover at job setup\nfunc newFS(ctx context.Context) (fs Interface) {\n    defer func() {\n        if r := recover(); r != nil {\n            log.Fatalf(\"GCS filesystem init failed: %v\", r)\n        }\n    }()\n    return gcs.New(ctx)\n}","preventionTips":["Run 'gcloud auth application-default login' in dev environments","Point GOOGLE_APPLICATION_CREDENTIALS at an existing, valid key file","Ensure Dataflow worker service account has cloud-platform scope","In containers/non-GCP hosts, emulate or reach the metadata server only if using metadata creds"],"tags":["go","gcs","credentials","google-cloud-storage"],"backgroundTag":"missing-credentials","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}