apache/beam · error · Exception

PubSub I/O is only available in streaming mode (use the…

Error message

PubSub I/O is only available in streaming mode (use the --streaming flag).

What it means

The DirectRunner's PubSub override only works in streaming mode: ReadFromPubSub requires an unbounded source and streaming execution, so in batch mode the runner raises Exception during transform replacement. The message tells you the exact flag needed.

Solutions

Add --streaming (StandardOptions.view_as(StandardOptions).streaming = True) when running PubSub pipelines on the DirectRunner.
Switch to a runner that supports the desired mode, e.g. DataflowRunner with --streaming.
Remove/replace the PubSub read in batch-only pipelines with a bounded source (e.g. Create or textio).

Example fix

// before
p = beam.Pipeline(runner='DirectRunner')  # batch
_ = p | beam.io.ReadFromPubSub(topic='projects/p/topics/t')
// after
options = PipelineOptions(['--streaming'])
p = beam.Pipeline(runner='DirectRunner', options=options)
_ = p | beam.io.ReadFromPubSub(topic='projects/p/topics/t')

Defensive patterns

Strategy: validation

Validate before calling

from apache_beam.options.pipeline_options import StandardOptions
opts = PipelineOptions(argv)
if uses_pubsub_io(pipeline) and not opts.view_as(StandardOptions).streaming:
    raise ValueError('PubSub reads require --streaming')

Try / catch

try:
    result = pipeline.run()
except Exception as e:
    if 'streaming mode' in str(e):
        options.view_as(StandardOptions).streaming = True  # rerun with --streaming
    else:
        raise

Prevention

Always pass --streaming for pipelines with unbounded sources in local tests.
Keep batch and streaming pipeline entry points separate.
Flag pipelines containing ReadFromPubSub as streaming-only in your tooling.

When it happens

Trigger: Running a pipeline with the DirectRunner that reads from PubSub via beam.io.ReadFromPubSub while StandardOptions.streaming is not set (no --streaming flag).

Common situations: Developing a streaming pipeline locally and forgetting --streaming; batch test harness accidentally including a PubSub read.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/892a4379b4bd913c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/direct/direct_runner.py:536

  def expand(self, pvalue):
    # This is handled as a native transform.
    return PCollection(self.pipeline, is_bounded=self._source.is_bounded())


def _get_pubsub_transform_overrides(pipeline_options):
  from apache_beam.io.gcp import pubsub as beam_pubsub
  from apache_beam.pipeline import PTransformOverride

  class ReadFromPubSubOverride(PTransformOverride):
    def matches(self, applied_ptransform):
      return isinstance(
          applied_ptransform.transform, beam_pubsub.ReadFromPubSub)

    def get_replacement_transform_for_applied_ptransform(
        self, applied_ptransform):
      if not pipeline_options.view_as(StandardOptions).streaming:
        raise Exception(
            'PubSub I/O is only available in streaming mode '
            '(use the --streaming flag).')
      return _DirectReadFromPubSub(applied_ptransform.transform._source)

  # WriteToPubSub no longer needs an override - it works by default for both
  # batch and streaming
  return [ReadFromPubSubOverride()]


class BundleBasedDirectRunner(PipelineRunner):
  """Executes a single pipeline on the local machine."""
  @staticmethod
  def is_fnapi_compatible():
    return False

  def run_pipeline(self, pipeline, options):
    """Execute the entire pipeline and returns an DirectPipelineResult."""

View on GitHub (pinned to 12126d8942)