What causes "Expected distinct components: ${id}" in apache/beam?

Rename transforms with unique labels in each pipeline (apply(PTransform, ...).apply with distinct names) so component ids differ.

apache/beam · error · Error

Expected distinct components

Error message

Expected distinct components: ${id}

What it means

mergeComponents() merges per-pipeline component maps (transforms, pcollections, etc.) into the combined allPipelines proto. If an id already exists in the destination with a different proto, it throws 'Expected distinct components: <id>' to prevent silently overwriting or aliasing incompatible components across pipelines.

Solutions

Rename transforms with unique labels in each pipeline (apply(PTransform, ...).apply with distinct names) so component ids differ.
Investigate the duplicate component id and why its protos differ between pipelines.
Remove the deepEqual assert call if near-equal protos should be tolerated, or normalize ids before merging.

Example fix

// before
const p1 = pipeline.apply("Sum", sum); // in two pipelines, same default ids
// after
pipeline1.apply("Sum-A", sum); pipeline2.apply("Sum-B", sum);

Defensive patterns

Strategy: validation

Validate before calling

function findDuplicateIds(pipelines) {
  const seen = new Map(); const dupes = [];
  for (const p of pipelines) for (const id of Object.keys(p.components.transforms)) {
    if (seen.has(id)) dupes.push(id); else seen.set(id, true);
  }
  return dupes;
}

Type guard

null

Try / catch

try {
  await multiRunner.runAsync();
} catch (e) {
  if (e.message.startsWith("Expected distinct components:")) {
    console.error("Rename transforms to unique labels across merged pipelines:", e.message);
  } else throw e;
}

Prevention

Give transforms explicit unique names in each pipeline
Check for duplicate auto-generated component ids before merging
Avoid identical default transform names across merged pipelines

When it happens

Trigger: Merging two or more pipelines into MultiPipelineRunner where two pipelines define the same component id (transform/pcollection/coder name) with different protos.

Common situations: Two pipelines generated by code that reuses default transform names (e.g. both contain a transform with the same auto-generated id) while being merged in a multi-pipeline test run.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/62b5607352f38e91. Report an issue: GitHub.

Appendix: source

Thrown at sdks/typescript/src/apache_beam/testing/multi_pipeline_runner.ts:144

      return "namespace_" + this.counter + "_";
    } finally {
      this.counter += 1;
    }
  }

  mergePipeline(pipeline: runnerApi.Pipeline) {
    if (this.allPipelines === undefined) {
      this.allPipelines = runnerApi.Pipeline.create({
        components: runnerApi.Components.create({}),
      });
    }
    function mergeComponents(src, dest) {
      for (const [id, proto] of Object.entries(src)) {
        if (dest[id] === undefined) {
          dest[id] = proto;
        } else if (dest[id] != proto) {
          require("assert").deepEqual(dest[id], proto);
          throw new Error("Expected distinct components: " + id);
        }
      }
    }
    mergeComponents(
      pipeline.components?.transforms,
      this.allPipelines.components?.transforms,
    );
    mergeComponents(
      pipeline.components?.pcollections,
      this.allPipelines.components?.pcollections,
    );
    mergeComponents(
      pipeline.components?.coders,
      this.allPipelines.components?.coders,
    );
    mergeComponents(
      pipeline.components?.windowingStrategies,
      this.allPipelines.components?.windowingStrategies,

View on GitHub (pinned to 12126d8942)