apache/beam · error · RuntimeException

Schema must be set for table " + tableReference + " when wri

Error message

Schema must be set for table " + tableReference + " when writing TableRows using Storage API and using a create disposition of CREATE_IF_NEEDED.

What it means

RuntimeException thrown by StorageApiDynamicDestinationsTableRow.TableRowConverter when writing TableRows with the Storage API and CreateDisposition.CREATE_IF_NEEDED, but no schema was provided and the table does not exist. TableRows are schema-less, so BigQueryIO requires an explicit schema to create the table.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiDynamicDestinationsTableRow.java:193

        DestinationT destination,
        DatasetService datasetService,
        @Nullable TableSchema localTableSchema)
        throws Exception {
      this.tableReference = getTable(destination).getTableReference();
      if (localTableSchema == null) {
        // If the table already exists, then try and fetch the schema from the existing
        // table.
        localTableSchema = SCHEMA_CACHE.getSchema(tableReference, datasetService);
        if (localTableSchema == null) {
          if (createDisposition == CreateDisposition.CREATE_NEVER) {
            throw new RuntimeException(
                "BigQuery table "
                    + tableReference
                    + " not found. If you wanted to "
                    + "automatically create the table, set the create disposition to CREATE_IF_NEEDED and specify a "
                    + "schema.");
          } else {
            throw new RuntimeException(
                "Schema must be set for table "
                    + tableReference
                    + " when writing TableRows using Storage API and "
                    + "using a create disposition of CREATE_IF_NEEDED.");
          }
        }
      } else {
        // Make sure we register this schema with the cache, unless there's already a more
        // up-to-date schema.
        localTableSchema =
            MoreObjects.firstNonNull(
                SCHEMA_CACHE.putSchemaIfAbsent(tableReference, localTableSchema), localTableSchema);
      }
      this.tableSchema = localTableSchema;
      this.protoTableSchema = TableRowToStorageApiProto.schemaToProtoTableSchema(tableSchema);
      this.getSchemaHash =
          Suppliers.memoize(() -> TableRowToStorageApiProto.tableSchemaHash(this.protoTableSchema));
      schemaInformation =

View on GitHub (pinned to 12126d8942)

Solutions

1. Add .withSchema(Schema or TableSchema) to the BigQueryIO write transform
2. Pre-create the destination table with the desired schema so the converter can fetch it
3. For DynamicDestinations, ensure getSchema(destination) returns a non-null schema for every destination

Example fix

// before
BigQueryIO.writeTableRows().to(tableSpec).withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
// after
BigQueryIO.writeTableRows().to(tableSpec)
  .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
  .withSchema(new TableSchema().setFields(fields));

if (createDisposition == CreateDisposition.CREATE_IF_NEEDED && writeSchema == null && bigquery.getTable(tableId) == null) {
  throw new IllegalArgumentException("Provide withSchema(...) for STORAGE_API TableRow writes");
}

try {
  result = writeResult;
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("Schema must be set for table")) {
    // rebuild the transform with .withSchema(...) and resubmit
  } else { throw e; }
}

When it happens

Trigger: BigQueryIO.writeTableRows().withCreateDisposition(CREATE_IF_NEEDED) with no .withSchema(...) call and the destination table absent, so dynamicDestinations.getSchema returns null.

Common situations: Switching from Avro/TableRow-based writes (which carry schema) to raw TableRows and forgetting withSchema; relying on a table that was later deleted; dynamic destinations where getSchema() returns null for a new destination.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Failed to patch table schema.
• We have observed a row of size %s bytes exceeding the BigQue
• Exception while trying to retrieve schema
• Failed to flush elements on window expiration!
• BigQuery table " + tableReference + " not found. If you want

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/dace97842dfdfa65. Report an issue: GitHub.