apache/beam · error · java.lang.IllegalArgumentException

SQL can only run over PCollections that have schemas.

Error message

SQL can only run over PCollections that have schemas.

What it means

BeamPCollectionTable wraps an existing PCollection so SQL can query it as a table, but SQL requires a schema to map columns. The constructor throws IllegalArgumentException if the input PCollection does not have a schema attached.

Solutions

Convert the PCollection to Rows and apply a schema: pcoll.apply(Convert.toRows(...)).setSchema(...)
Use BeamSQL's built-in connectors (e.g. BeamTextTable) instead of wrapping a raw PCollection
Ensure the element type is a class with schema-inferable fields (annotated @DefaultSchema or a simple POJO) and call setRowSchema/setSchema

Example fix

// before
pcoll.apply(ParDo.of(new ParseFn()));
sqlContext.registerTable("t", new BeamPCollectionTable<>(pcoll));
// after
PCollection<Row> rows = pcoll.apply(...).setRowSchema(mySchema);
sqlContext.registerTable("t", new BeamPCollectionTable<>(rows));

Defensive patterns

Strategy: validation

Validate before calling

if (!pcollection.hasSchema()) { throw new IllegalArgumentException("apply a schema before registering table"); }

Type guard

boolean isSqlReady(PCollection<?> p) { return p.hasSchema(); }

Try / catch

try { table = new BeamPCollectionTable<>(pcoll); } catch (IllegalArgumentException e) { pcoll = pcoll.apply(Convert.toRows(...)).setRowSchema(schema); }

Prevention

Always call setSchema/setRowSchema on PCollections before SQL registration
Use typed POJOs with @DefaultSchema annotations
Prefer built-in table connectors for raw text sources

When it happens

Trigger: Calling sqlContext.registerTable (or constructing BeamPCollectionTable directly) with a PCollection<POJO>/PCollection<Row> created without apply(Schema) — e.g. a PCollection<String> from a text source registered as-is.

Common situations: Registering a raw text/JSON PCollection without converting to typed rows or applying a schema; using Java 8 lambdas producing untyped types that Beam cannot infer schemas for.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/fc527dab8342747f. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java:37

import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.schemas.transforms.Convert;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.Row;

/**
 * {@code BeamPCollectionTable} converts a {@code PCollection<Row>} as a virtual table, then a
 * downstream query can query directly.
 */
public class BeamPCollectionTable<InputT> extends SchemaBaseBeamTable {
  private transient PCollection<InputT> upstream;

  public BeamPCollectionTable(PCollection<InputT> upstream) {
    super(upstream.getSchema());
    if (!upstream.hasSchema()) {
      throw new IllegalArgumentException("SQL can only run over PCollections that have schemas.");
    }
    this.upstream = upstream;
  }

  @Override
  public PCollection.IsBounded isBounded() {
    return upstream.isBounded();
  }

  @Override
  public PCollection<Row> buildIOReader(PBegin begin) {
    assert begin.getPipeline() == upstream.getPipeline();
    return upstream.apply(Convert.toRows());
  }

  @Override
  public POutput buildIOWriter(PCollection<Row> input) {
    throw new IllegalArgumentException("cannot use [BeamPCollectionTable] as target");

View on GitHub (pinned to 12126d8942)