apache/beam · error · RuntimeException

Unable to determine the owner of the default bucket at gs://

Error message

Unable to determine the owner of the default bucket at gs://${bucketName}

What it means

After ensuring the default bucket exists, tryCreateDefaultBucketWithPrefix calls GcsUtil.bucketOwner to verify ownership; IOException is wrapped as RuntimeException 'Unable to determine the owner of the default bucket at gs://<bucketName>'. The library could not read bucket metadata to confirm the bucket belongs to the current project.

Source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java:506

      try {
        gcsOptions.getGcsUtil().createBucket(gcsOptions.getProject(), bucket);
      } catch (FileAlreadyExistsException e) {
        LOG.debug("Bucket '{}'' already exists, verifying access.", bucketName);
      } catch (IOException e) {
        throw new RuntimeException("Unable create default bucket.", e);
      }

      // Once the bucket is expected to exist, verify that it is correctly owned
      // by the project executing the job.
      try {
        long owner = gcsOptions.getGcsUtil().bucketOwner(GcsPath.fromComponents(bucketName, ""));
        checkArgument(
            owner == projectNumber,
            "Bucket owner does not match the project from --project:" + " %s vs. %s",
            owner,
            projectNumber);
      } catch (IOException e) {
        throw new RuntimeException(
            "Unable to determine the owner of the default bucket at gs://" + bucketName, e);
      }
      return "gs://" + bucketName + "/temp/";
    }

    /**
     * Returns the project number or throws an exception if the project does not exist or has other
     * access exceptions.
     */
    private static long getProjectNumber(String projectId, CloudResourceManager crmClient)
        throws IOException {
      return getProjectNumber(
          projectId,
          crmClient,
          BackOffAdapter.toGcpBackOff(BACKOFF_FACTORY.backoff()),
          Sleeper.DEFAULT);
    }

View on GitHub (pinned to 12126d8942)

Solutions

1. Grant storage.buckets.get (roles/storage.legacyBucketOwner or roles/storage.objectAdmin+bucket get) to the identity
2. Set an explicit --tempLocation bucket you own to skip the ownership probe
3. Retry on transient failures; check GCS status dashboard
4. Confirm the bucket was not removed concurrently

Example fix

// before
// default bucket, service account lacks bucket get
// after
gsutil iam ch serviceAccount:sa@project.iam.gserviceaccount.com:roles/storage.admin gs://bucket

// pre-check ownership with gsutil or GCS API before relying on default bucket
// gsutil bucket-policy-only / get-IAM-policy, or Storage.buckets.get(bucket) via client

try {
  String b = gcpOptions.getDefaultBucketName();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("Unable to determine the owner")) {
    // retry once after backoff, then fall back to explicit tempLocation
    gcpOptions.setTempLocation("gs://owned-bucket/temp/");
  } else throw e;
}

When it happens

Trigger: Immediately after createBucket (or when it already existed), bucketOwner(GcsPath) throws IOException: storage.buckets.get permission denied, bucket deleted in a race, requester-pays bucket, or transient API failure.

Common situations: Service account can create but not view buckets (missing storage.buckets.get); bucket created by another project with the same derived name; transient GCS 5xx.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• Unable create default bucket.
• Error constructing default value for gcpTempLocation: tempLo
• Unable to get project number
• The key '%s' in GCS custom audit entries exceeds the %d-char
• The value '%s' in GCS custom audit entries exceeds the %d-ch

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f7e7f2136748dac1. Report an issue: GitHub.