apache/beam · error · IllegalArgumentException

Output of composite transform

Error message

Output of composite transform [%s] contains a %s produced by it. Only primitive transforms are permitted to produce %ss.%n    Outputs: %s%n    Other Producers: %s%n    Components: %s

What it means

Beam's TransformHierarchy validates that when a composite transform sets its output, the produced PCollection(s) are not also listed as outputs of the composite itself. Only primitive (leaf) transforms may produce PCollections. This invariant catches malformed pipeline graph construction where a composite both expands into children and claims their outputs.

Solutions

Fix the composite PTransform so apply()/expand() returns PCollections produced by primitive child transforms, not by the composite itself.
Do not call applyOutput/producer bookkeeping APIs directly on composite nodes; let expansion assign producers.
Check for custom RunnerApi pipeline expansion or overrides that remap PCollection producers.
Print the transform graph (e.g. with --experiments or graph dumps) and verify producer attribution of the collections listed in the message.

Example fix

// before (composite directly returns input unchanged marked as own output)
PCollection<T> expand(PCollection<T> in) { return in.apply("bad", this); }

// after: delegate to a primitive child
PCollection<T> expand(PCollection<T> in) { return in.apply("good", MapElements.via(...)); }

Defensive patterns

Strategy: validation

Validate before calling

// In tests, expand the composite and assert outputs are produced by primitive children
PCollectionList out = pipeline.apply(myComposite);
assertNotEquals(myComposite, out.get(0).getProducer());

Try / catch

try { pipeline.apply("t", composite).apply(next); }
catch (IllegalArgumentException e) {
  if (e.getMessage().contains("Only primitive transforms are permitted")) {
    throw new IllegalStateException("Composite transform misconfigured: " + e.getMessage());
  } throw e;
}

Prevention

Only return child-produced PCollections from expand()
Don't touch internal TransformHierarchy producer APIs from composites
Test custom composites with a direct runner before production

When it happens

Trigger: Triggered in setOutput during pipeline expansion when the output PCollections of a composite have a producer node other than the composite's own child — i.e. the output was produced by the composite while other producers were recorded for the same collections.

Common situations: Custom composite PTransform implementations that override expand() incorrectly and reuse/reassign output PCollections; runners or test harnesses building the transform tree manually and misattributing producers; using internal runner APIs to graft subgraphs.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/847636cb85cb7ebe. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java:429

      checkState(
          this.outputs == null, "Tried to specify more than one output for %s", getFullName());
      checkNotNull(output, "Tried to set the output of %s to null", getFullName());
      this.outputs = PValues.expandOutput(output);

      // Validate that a primitive transform produces only primitive output, and a composite
      // transform does not produce primitive output.
      Set<Node> outputProducers = new HashSet<>();
      for (PCollection<?> outputValue : PValues.expandOutput(output).values()) {
        outputProducers.add(getProducer(outputValue));
      }
      if (outputProducers.contains(this) && (!parts.isEmpty() || outputProducers.size() > 1)) {
        Set<String> otherProducerNames = new HashSet<>();
        for (Node outputProducer : outputProducers) {
          if (outputProducer != this) {
            otherProducerNames.add(outputProducer.getFullName());
          }
        }
        throw new IllegalArgumentException(
            String.format(
                "Output of composite transform [%s] contains a %s produced by it. "
                    + "Only primitive transforms are permitted to produce %ss."
                    + "%n    Outputs: %s"
                    + "%n    Other Producers: %s"
                    + "%n    Components: %s",
                getFullName(),
                PCollection.class.getSimpleName(),
                PCollection.class.getSimpleName(),
                output.expand(),
                otherProducerNames,
                parts));
      }
    }

    /**
     * Replaces each value in {@code originalToReplacement} present in this {@link Node Node's}
     * outputs with the key that maps to that value.

View on GitHub (pinned to 12126d8942)