apache/beam · critical · RuntimeException

Failed to create subscription to topic %s on project %s: %s

Error message

Failed to create subscription to topic %s on project %s: %s

What it means

PubsubUnboundedSource's subscription auto-creation path wraps any Exception during PubsubClient.createSubscription into RuntimeException("Failed to create subscription to topic %s on project %s: %s", e). It is thrown when reading from a topic path requires Beam to create a scratch subscription and the Pubsub API call fails. The formatted message carries the topic path, project path, and underlying error message.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java:1475

          topicPath);
      projectPath = PubsubClient.projectPathFromId(projectId);
    }

    try {
      try (PubsubClient pubsubClient =
          pubsubFactory.newClient(
              timestampAttribute, idAttribute, options.as(PubsubOptions.class))) {
        SubscriptionPath subscriptionPath =
            pubsubClient.createRandomSubscription(projectPath, topicPath, DEAULT_ACK_TIMEOUT_SEC);
        LOG.warn(
            "Created subscription {} to topic {}."
                + " Note this subscription WILL NOT be deleted when the pipeline terminates",
            subscriptionPath,
            topic);
        return subscriptionPath;
      }
    } catch (Exception e) {
      throw new RuntimeException(
          String.format(
              "Failed to create subscription to topic %s on project %s: %s",
              topicPath, projectPath, e.getMessage()),
          e);
    }
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Pre-create the subscription and pass it via fromSubscription(...) instead of relying on auto-creation.
2. Grant the pipeline service account roles/pubsub.editor (or subscriptions.create + topic attach) on the target project.
3. Verify the topic exists and the topic path/project strings are correct.
4. Check the chained cause message for quota/API-specific errors and address those (quota increase, retry later).

Example fix

// before
PubsubIO.readMessages().fromTopic("projects/myproj/topics/mytopic") // auto-create may fail
// after
// gcloud pubsub subscriptions create mytopic-sub --topic=mytopic --topic-project=myproj
PubsubIO.readMessages().fromSubscription("projects/myproj/subscriptions/mytopic-sub");

gcloud pubsub topics describe <topicPath>
gcloud projects get-iam-policy <project> --flatten="bindings[].members" --filter="bindings.members:<svc-account>"

null

try { read.fromTopic(topic); } catch (RuntimeException e) { LOG.error("subscription creation failed: {}", e.getCause()); /* fall back to pre-created subscription */ }

When it happens

Trigger: Using PubsubIO.read().topic(topicPath) (no explicit subscription) where Beam must create a subscription, and creation fails: permission denied (needs pubsub.subscriptions.create on the project), invalid topic path, topic doesn't exist, or API/quota errors.

Common situations: Service account lacking Pubsub Editor/Subscriber+Editor roles; reading from a topic in another project without cross-project permissions; deleted topic still referenced; transient Pubsub API outage during pipeline startup.

Related errors

• Unable create default bucket.
• Unable to determine the owner of the default bucket at gs://
• Unable to get project number
• Pubsub topic '%s' does not exist.
• Received message missing publishTime

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/122e2fa46378cd5f. Report an issue: GitHub.