apache/beam · error · RuntimeException

Unable to verify project with ID ${projectId}

Error message

Unable to verify project with ID ${projectId}

What it means

GcpOptions.getDefaultBucketName() verifies the project by looking up its project number via the Cloud Resource Manager API; an IOException there is wrapped as RuntimeException 'Unable to verify project with ID <projectId>'. The project ID was non-empty but could not be confirmed to exist/be accessible.

Source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java:441

      } catch (Exception e) {
        LOG.warn("Failed to access bucket for gcpTempLocation: {}", tempLocation, e);
      }
      return false;
    }

    @VisibleForTesting
    static ImmutableList<String> getDefaultBucketNameStubs(
        PipelineOptions options, CloudResourceManager crmClient, String bucketNamePrefix) {
      GcsOptions gcsOptions = options.as(GcsOptions.class);

      final String projectId = gcsOptions.getProject();
      checkArgument(!isNullOrEmpty(projectId), "--project is a required option.");

      long projectNumber = 0L;
      try {
        projectNumber = getProjectNumber(projectId, crmClient);
      } catch (IOException e) {
        throw new RuntimeException("Unable to verify project with ID " + projectId, e);
      }

      String region = DEFAULT_REGION;
      if (!isNullOrEmpty(gcsOptions.getZone())) {
        region = getRegionFromZone(gcsOptions.getZone());
      }

      return ImmutableList.of(bucketNamePrefix, region, String.valueOf(projectNumber));
    }

    /**
     * Creates a default bucket or verifies the existence and proper access control of an existing
     * default bucket. Returns the location if successful.
     */
    @VisibleForTesting
    static String tryCreateDefaultBucket(PipelineOptions options, CloudResourceManager crmClient) {
      return tryCreateDefaultBucketWithPrefix(options, crmClient, "dataflow-staging");
    }

View on GitHub (pinned to 12126d8942)

Solutions

  1. Verify the project ID with `gcloud projects describe <projectId>`
  2. Enable the Cloud Resource Manager API in the project
  3. Grant the caller resourcemanager.projects.get (e.g. roles/viewer)
  4. Check credentials/network so the CRM request can succeed

Example fix

// before
--project=my-projct // typo
// after
--project=my-project // verified via gcloud projects describe
Defensive patterns

Strategy: validation

Validate before calling

// ensure the project is resolvable before building default bucket
String projectId = options.as(GcpOptions.class).getProject();
if (projectId == null || projectId.isEmpty()) throw new IllegalArgumentException("--project required");
// optionally verify via gcloud or CRM API call before launching

Try / catch

try {
  String bucket = options.as(GcpOptions.class).getDefaultBucketName();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Unable to verify project")) {
    throw new IllegalStateException("Check project ID, CRM API, and resourcemanager.projects.get permission", e);
  }
  throw e;
}

Prevention

When it happens

Trigger: Calling GcpOptions.getDefaultBucketName() (via bucketNameStubs) with a --project value for which getProjectNumber() throws IOException: nonexistent project ID, no CRM API permission, API not enabled, or network/auth failure.

Common situations: Typo in project ID; using project name instead of ID; service account lacking resourcemanager.projects.get; cloudresourcemanager.googleapis.com disabled; running off-GCP without credentials.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a9f1f105358d3615. Report an issue: GitHub.