apache/beam · error · IllegalArgumentException

PCollections come from different Pipelines

Error message

PCollections come from different Pipelines

What it means

PCollectionList.and(PCollection) requires every added PCollection to belong to the same Pipeline as the list. Beam PCollections cannot be mixed across Pipelines, so a mismatch throws IllegalArgumentException immediately.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionList.java:113

    Iterator<PCollection<T>> pcsIter = pcs.iterator();
    if (!pcsIter.hasNext()) {
      throw new IllegalArgumentException(
          "must either have a non-empty list of PCollections, "
              + "or must first call empty(Pipeline)");
    }
    return new PCollectionList<T>(pcsIter.next().getPipeline()).and(pcs);
  }

  /**
   * Returns a new {@link PCollectionList} that has all the {@link PCollection PCollections} of this
   * {@link PCollectionList} plus the given {@link PCollection} appended to the end.
   *
   * <p>All the {@link PCollection PCollections} in the resulting {@link PCollectionList} must be
   * part of the same {@link Pipeline}.
   */
  public PCollectionList<T> and(PCollection<T> pc) {
    if (pc.getPipeline() != pipeline) {
      throw new IllegalArgumentException("PCollections come from different Pipelines");
    }
    return new PCollectionList<>(
        pipeline,
        ImmutableList.<TaggedPValue>builder()
            .addAll(pcollections)
            .add(TaggedPValue.of(new TupleTag<T>(Integer.toString(pcollections.size())), pc))
            .build());
  }

  /**
   * Returns a new {@link PCollectionList} that has all the {@link PCollection PCollections} of this
   * {@link PCollectionList} plus the given {@link PCollection PCollections} appended to the end, in
   * order.
   *
   * <p>All the {@link PCollection PCollections} in the resulting {@link PCollectionList} must be
   * part of the same {@link Pipeline}.
   */
  public PCollectionList<T> and(Iterable<PCollection<T>> pcs) {

View on GitHub (pinned to 12126d8942)

Solutions

1. Ensure all PCollections come from the same Pipeline instance before combining
2. Pass the Pipeline into helper methods so they create PCollections from the correct Pipeline
3. Re-create the downstream transforms on the same Pipeline instead of crossing pipelines

Example fix

// before
PCollection<T> other = otherPipeline.apply(...);
list.and(other); // IllegalArgumentException
// after
PCollection<T> other = samePipeline.apply(...);
list.and(other);

if (pc.getPipeline() != list.getPipeline()) { throw new IllegalArgumentException("wrong pipeline"); }

boolean samePipeline(PCollection<?> pc, Pipeline p) { return pc.getPipeline() == p; }

try { return list.and(pc); } catch (IllegalArgumentException e) { throw new IllegalStateException("PCollection from different Pipeline", e); }

When it happens

Trigger: Appending a PCollection produced by a different Pipeline object (identity comparison pc.getPipeline() != pipeline) to an existing PCollectionList.

Common situations: Building inputs from two separately-constructed Pipelines; accidentally re-creating a Pipeline in a helper method and mixing its outputs with the main pipeline's; combining test pipeline outputs with production ones.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• PCollections come from different Pipelines
• PCollections come from different Pipelines
• The pipeline has not been run.
• The pipeline contains abandoned PTransform(s).
• ApproximateUnique.PerKey needs an estimation error between 1

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8fd71027ac299354. Report an issue: GitHub.