apache/beam · error · IllegalArgumentException

Could not resolve GCP project ID%s. Please specify 'project'

Error message

Could not resolve GCP project ID%s. Please specify 'project' in the secret spec, set GOOGLE_CLOUD_PROJECT environment variable, or configure Application Default Credentials.

What it means

GcpSecret.resolveGcpProjectId resolves the GCP project when the spec lacks 'project': it tries the GOOGLE_CLOUD_PROJECT env var, Application Default Credentials, and ServiceOptions via reflection. If all fail it throws IllegalArgumentException instructing the user how to specify the project. The context suffix names which secret was being resolved.

Source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcpSecret.java:119

    if (!Strings.isNullOrEmpty(envProject)) {
      return Preconditions.checkNotNull(envProject);
    }
    envProject = System.getenv("GCP_PROJECT");
    if (!Strings.isNullOrEmpty(envProject)) {
      return Preconditions.checkNotNull(envProject);
    }
    try {
      Class<?> clazz = Class.forName("com.google.cloud.ServiceOptions");
      java.lang.reflect.Method method = clazz.getMethod("getDefaultProjectId");
      @SuppressWarnings("nullness")
      Object result = method.invoke(null);
      if (result != null && !Strings.isNullOrEmpty(result.toString())) {
        return result.toString();
      }
    } catch (Throwable e) {
      LOG.debug("Could not resolve GCP project via ServiceOptions reflection", e);
    }
    throw new IllegalArgumentException(
        String.format(
            "Could not resolve GCP project ID%s. "
                + "Please specify 'project' in the secret spec, set GOOGLE_CLOUD_PROJECT environment variable, "
                + "or configure Application Default Credentials.",
            context != null ? " for " + context : ""));
  }

  /**
   * Returns the secret as a byte array. Assumes that the current active service account has
   * permissions to read the secret.
   *
   * @return The secret as a byte array.
   */
  @Override
  public byte[] getSecretBytes() {
    try (SecretManagerServiceClient client = SecretManagerServiceClient.create()) {
      SecretVersionName secretVersionName = SecretVersionName.parse(versionName);
      AccessSecretVersionResponse response = client.accessSecretVersion(secretVersionName);

View on GitHub (pinned to 12126d8942)

Solutions

Add "project": "<project-id>" to the secret spec
Set the GOOGLE_CLOUD_PROJECT environment variable
Run gcloud auth application-default login to establish ADC with a quota project
When running on GCP (GCE/GKE/Cloud Run), attach a service account so the metadata server supplies the project

Example fix

// before
GcpSecret.fromMap(Map.of("name","my-secret"));
// after
GcpSecret.fromMap(Map.of("name","my-secret","project","my-project"));

Defensive patterns

Strategy: validation

Validate before calling

String project = spec.get("project") != null ? spec.get("project") : System.getenv("GOOGLE_CLOUD_PROJECT");
if (project == null) throw new IllegalArgumentException("Set 'project' in spec or GOOGLE_CLOUD_PROJECT before resolving secrets");

Try / catch

try { GcpSecret.fromMap(spec); }
catch (IllegalArgumentException e) {
  if (e.getMessage().contains("Could not resolve GCP project")) {
    LOG.error("Set GOOGLE_CLOUD_PROJECT or add 'project' to the spec");
  }
  throw e;
}

Prevention

Always set 'project' explicitly in secret specs for non-GCP environments
Set GOOGLE_CLOUD_PROJECT in CI and container images
Run gcloud auth application-default login locally before running pipelines

When it happens

Trigger: Calling GcpSecret.fromMap without 'project' in the spec on a machine with no GOOGLE_CLOUD_PROJECT env var, no gcloud ADC setup, and no ServiceOptions default project available.

Common situations: Running locally before gcloud auth application-default login; running in a non-GCP environment (CI, laptop) without explicit project config; containers lacking metadata-server access.

Understand the failure class

Background: "missing required config value" errors: why libraries refuse to start when a configuration key is empty, unset, or blank — this error's family across 48 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9cc1b01ebb763c42. Report an issue: GitHub.