apache/beam · error · ValueError

PubSub topic must be in the form "projects/<project>/topics/

Error message

PubSub topic must be in the form "projects/<project>/topics/<topic>" (got %r).

What it means

parse_topic validates that a Pub/Sub topic string matches 'projects/([^/]+)/topics/(.+)' before constructing PubSub sinks/sources. A ValueError is raised when the string does not follow the fully-qualified projects/<project>/topics/<topic> form.

Source

Thrown at sdks/python/apache_beam/io/gcp/pubsub.py:489

        'with_attributes': DisplayDataItem(
            True, label='With Attributes').drop_if_none(),
        'timestamp_attribute': DisplayDataItem(
            self.timestamp_attribute, label='Timestamp Attribute'),
        'publish_with_ordering_key': DisplayDataItem(
            self.publish_with_ordering_key,
            label='Publish With Ordering Key').drop_if_none(),
    }


PROJECT_ID_REGEXP = '[a-z][-a-z0-9:.]{4,61}[a-z0-9]'
SUBSCRIPTION_REGEXP = 'projects/([^/]+)/subscriptions/(.+)'
TOPIC_REGEXP = 'projects/([^/]+)/topics/(.+)'


def parse_topic(full_topic: str) -> tuple[str, str]:
  match = re.match(TOPIC_REGEXP, full_topic)
  if not match:
    raise ValueError(
        'PubSub topic must be in the form "projects/<project>/topics'
        '/<topic>" (got %r).' % full_topic)
  project, topic_name = match.group(1), match.group(2)
  if not re.match(PROJECT_ID_REGEXP, project):
    raise ValueError('Invalid PubSub project name: %r.' % project)
  return project, topic_name


def parse_subscription(full_subscription):
  match = re.match(SUBSCRIPTION_REGEXP, full_subscription)
  if not match:
    raise ValueError(
        'PubSub subscription must be in the form "projects/<project>'
        '/subscriptions/<subscription>" (got %r).' % full_subscription)
  project, subscription_name = match.group(1), match.group(2)
  if not re.match(PROJECT_ID_REGEXP, project):
    raise ValueError('Invalid PubSub project name: %r.' % project)
  return project, subscription_name

View on GitHub (pinned to 12126d8942)

Solutions

1. Pass the fully-qualified form: 'projects/my-project/topics/my-topic'
2. Format the string programmatically from project and topic variables: f'projects/{project}/topics/{topic}'
3. Check for stray whitespace or a leading/trailing slash that breaks the regexp match

Example fix

// before
WriteToPubSub(topic='my-topic')
// after
WriteToPubSub(topic='projects/my-gcp-project/topics/my-topic')

import re
assert re.match(r'projects/([^/]+)/topics/(.+)', topic), f'bad topic: {topic!r}'

try:
    pcoll | beam.io.WriteToPubSub(topic=topic)
except ValueError as e:
    if 'PubSub topic must be in the form' in str(e):
        topic = f'projects/{PROJECT}/topics/{topic_name}'
        pcoll | beam.io.WriteToPubSub(topic=topic)
    else:
        raise

When it happens

Trigger: Calling WriteToPubSub/ReadFromPubSub (or parse_topic directly) with a bare topic name like 'my-topic', a URL like 'pubsub.googleapis.com/...', or a truncated path.

Common situations: Users pasting only the topic id from the GCP console; forgetting the project prefix; using subscription-form strings where a topic is expected.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• Either data (%r) or attributes (%r) must be set.
• Invalid PubSub project name: %r.
• PubSub subscription must be in the form "projects/<project>/
• Either a topic or subscription must be provided.
• Only one of topic or subscription should be provided.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6d4e5160006e6e83. Report an issue: GitHub.