apache/beam · error · ValueError

Could not resolve GCP project ID for secret

Error message

Could not resolve GCP project ID for secret '{secret_id}'. Please specify 'project' in the secret spec, set GOOGLE_CLOUD_PROJECT environment variable, or configure Application Default Credentials.

What it means

To build 'projects/<project>/secrets/<name>/versions/<version>', GcpSecret._parse_version_name must resolve a GCP project ID. It checks spec 'project', then GOOGLE_CLOUD_PROJECT and GCP_PROJECT env vars, then Application Default Credentials. If all fail, this ValueError is raised.

Solutions

1. Add 'project': '<gcp-project-id>' to the secret spec dict.
2. Export GOOGLE_CLOUD_PROJECT=<project-id> in the environment.
3. Set up Application Default Credentials (gcloud auth application-default login) so the project can be inferred.
4. Pass 'version_name' with the fully-qualified path to skip project resolution entirely.

Example fix

// before
secret = GcpSecret.from_dict({'name': 'db-pass'})
// after
secret = GcpSecret.from_dict({'name': 'db-pass', 'project': 'my-gcp-project'})

import os
project = spec.get('project') or os.environ.get('GOOGLE_CLOUD_PROJECT') or os.environ.get('GCP_PROJECT')
if not project:
    raise ValueError('No GCP project resolvable for secret; set GOOGLE_CLOUD_PROJECT')

def can_resolve_project(spec) -> bool:
    import os
    return bool(spec.get('project') or os.environ.get('GOOGLE_CLOUD_PROJECT') or os.environ.get('GCP_PROJECT'))

try:
    secret = GcpSecret.from_dict(spec)
except ValueError as e:
    raise ConfigError('Set GOOGLE_CLOUD_PROJECT or pass project in spec') from e

When it happens

Trigger: GcpSecret.from_dict({'name': 'db-pass'}) with no 'project' key, no GOOGLE_CLOUD_PROJECT/GCP_PROJECT env var set, and no ADC available (e.g. running locally without gcloud auth, or in a container without a service account).

Common situations: Local development without `gcloud auth application-default login`; CI runners lacking project metadata; Docker/K8s images without workload identity or env vars.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

• Both deidentification_template_name and…
• Datastore total statistics unavailable.
• deidentification_template_name or deidentification_config…
• Error creating Data Catalog client
• Failed to retrieve secret bytes for secret

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8987e35602db0261. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/utils/secret.py:248

    if not secret_id:
      raise ValueError("Secret name must be specified in secret spec.")

    # Resolve project ID from spec, environment variables, or Application Default Credentials
    project_id = (
        spec_dict.get("project") or os.environ.get("GOOGLE_CLOUD_PROJECT") or
        os.environ.get("GCP_PROJECT"))

    if not project_id:
      try:
        import google.auth
        _, project_id = google.auth.default()
      except Exception:
        pass

    version_id = spec_dict.get("version", "latest")

    if not project_id:
      raise ValueError(
          f"Could not resolve GCP project ID for secret '{secret_id}'. "
          "Please specify 'project' in the secret spec, set GOOGLE_CLOUD_PROJECT environment variable, "
          "or configure Application Default Credentials.")

    return f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"

  def get_secret_bytes(self) -> bytes:
    try:
      from google.cloud import secretmanager
      client = secretmanager.SecretManagerServiceClient()
      response = client.access_secret_version(
          request={"name": self._version_name})
      secret = response.payload.data
      return secret
    except Exception as e:
      raise RuntimeError(
          'Failed to retrieve secret bytes for secret '
          f'{self._version_name} with exception {e}')

View on GitHub (pinned to 12126d8942)