apache/beam · error · RuntimeError

A transform with label "%s" already exists in the pipeline.

Error message

A transform with label "%s" already exists in the pipeline. To apply a transform with a specified label, write pvalue | "label" >> transform or use the option "auto_unique_labels" to automatically generate unique transform labels. Note "auto_unique_labels" could cause data loss when updating a pipeline or reloading the job state. This is not recommended for streaming jobs.

What it means

Transform labels in a pipeline must be unique. When applying a transform whose label is already used, and the auto_unique_labels option is off, Beam raises RuntimeError explaining how to either give an explicit unique label via the | 'label' >> transform syntax or enable auto_unique_labels (with a data-loss caveat for streaming/update use).

Source

Thrown at sdks/python/apache_beam/pipeline.py:757

    if self._current_transform() is self._root_transform():
      alter_label_if_ipython(transform, pvalueish)

    full_label = '/'.join(
        [self._current_transform().full_label, transform.label]).lstrip('/')
    if full_label in self.applied_labels:
      auto_unique_labels = self._options.view_as(
          StandardOptions).auto_unique_labels
      if auto_unique_labels:
        # If auto_unique_labels is set, we will append a unique suffix to the
        # label to make it unique.
        logging.warning(
            'Using --auto_unique_labels could cause data loss when '
            'updating a pipeline or reloading the job state. '
            'This is not recommended for streaming jobs.')
        unique_label = self._generate_unique_label(transform)
        return self.apply(transform, pvalueish, unique_label)
      else:
        raise RuntimeError(
            'A transform with label "%s" already exists in the pipeline. '
            'To apply a transform with a specified label, write '
            'pvalue | "label" >> transform or use the option '
            '"auto_unique_labels" to automatically generate unique '
            'transform labels. Note "auto_unique_labels" '
            'could cause data loss when updating a pipeline or '
            'reloading the job state. This is not recommended for '
            'streaming jobs.' % full_label)
    self.applied_labels.add(full_label)

    if pvalueish is None:
      full_label = self._current_transform().full_label
      raise TypeCheckError(
          f'Transform "{full_label}" was applied to the output of '
          f'an object of type None.')

    pvalueish, inputs = transform._extract_input_pvalues(pvalueish)
    try:

View on GitHub (pinned to 12126d8942)

Solutions

1. Give each transform a unique label: pcoll | 'my_unique_label' >> MyTransform().
2. Enable the auto_unique_labels option to auto-generate unique labels (avoid for streaming/job-update).
3. Refactor loops to interpolate a distinct label per iteration.
4. Use fresh Pipeline objects instead of reusing one for multiple graph builds.

Example fix

// before
for i in range(2):
  pcoll = pcoll | beam.Map(lambda x: x + 1)  # duplicate label 'Map'
// after
for i in range(2):
  pcoll = pcoll | ('inc_%d' % i) >> beam.Map(lambda x: x + 1)

try:
    pcoll = pcoll | label >> transform
except RuntimeError as e:
    if 'already exists in the pipeline' in str(e):
        label = f'{label}_{uuid.uuid4().hex[:8]}'
        pcoll = pcoll | label >> transform

When it happens

Trigger: Applying two transforms that resolve to the same default label (e.g. two unnamed beam.Map of the same function, or two Create/Impulse nodes) in the same pipeline; re-applying a transform to multiple outputs without renaming.

Common situations: Loops building multiple similar steps with identical default labels; reusing a pipeline object for multiple runs; applying the same named transform twice ('read' twice).

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Transform node %r was not replaced as expected.
• You cannot turn on runtime_type_check and performance_runtim
• Start Bundle should not output any elements but got %s
• Finish Bundle should only output WindowedValue type but got
• Please specify InteractiveRunner when creating the Beam pipe

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2ff33dd3df2031f5. Report an issue: GitHub.