apache/beam · error · RuntimeException

BigQuery table " + tableReference + " not found. If you want

Error message

BigQuery table " + tableReference + " not found. If you wanted to automatically create the table, set the create disposition to CREATE_IF_NEEDED and specify a schema.

What it means

RuntimeException thrown by StorageApiDynamicDestinationsTableRow.TableRowConverter when the destination BigQuery table's schema cannot be fetched and the write's create disposition is CREATE_NEVER. Since the library is forbidden from creating the table, a missing table is fatal.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiDynamicDestinationsTableRow.java:186

    final com.google.cloud.bigquery.storage.v1.TableSchema protoTableSchema;
    final Supplier<byte[]> getSchemaHash;
    final TableRowToStorageApiProto.SchemaInformation schemaInformation;
    final Descriptor descriptor;
    final @Nullable Descriptor cdcDescriptor;

    TableRowConverter(
        DestinationT destination,
        DatasetService datasetService,
        @Nullable TableSchema localTableSchema)
        throws Exception {
      this.tableReference = getTable(destination).getTableReference();
      if (localTableSchema == null) {
        // If the table already exists, then try and fetch the schema from the existing
        // table.
        localTableSchema = SCHEMA_CACHE.getSchema(tableReference, datasetService);
        if (localTableSchema == null) {
          if (createDisposition == CreateDisposition.CREATE_NEVER) {
            throw new RuntimeException(
                "BigQuery table "
                    + tableReference
                    + " not found. If you wanted to "
                    + "automatically create the table, set the create disposition to CREATE_IF_NEEDED and specify a "
                    + "schema.");
          } else {
            throw new RuntimeException(
                "Schema must be set for table "
                    + tableReference
                    + " when writing TableRows using Storage API and "
                    + "using a create disposition of CREATE_IF_NEEDED.");
          }
        }
      } else {
        // Make sure we register this schema with the cache, unless there's already a more
        // up-to-date schema.
        localTableSchema =
            MoreObjects.firstNonNull(

View on GitHub (pinned to 12126d8942)

Solutions

1. Create the table in BigQuery (or fix the table reference spelling/project) before running the pipeline
2. Change the write to use CreateDisposition.CREATE_IF_NEEDED and supply a schema via withSchema(...)
3. Verify the credentials can access the table in the specified project
4. Use bq show or the console to confirm the fully-qualified tableReference exists

Example fix

// before
BigQueryIO.writeTableRows().to(tableSpec).withCreateDisposition(CreateDisposition.CREATE_NEVER)
// after
BigQueryIO.writeTableRows().to(tableSpec)
  .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
  .withSchema(tableSchema);

TableId id = TableId.of(project, dataset, table);
if (bigquery.getTable(id) == null) {
  throw new IllegalStateException("Table " + id + " does not exist; create it or use CREATE_IF_NEEDED");
}

try {
  pipeline.run().waitUntilFinish();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("BigQuery table") && e.getMessage().contains("not found")) {
    bigquery.create(TableInfo.newBuilder(tableId, TableDefinition.of(StandardTableDefinition.of(schema))).build());
    // resubmit the pipeline
  } else { throw e; }
}

When it happens

Trigger: Writing TableRows via BigQueryIO with Storage API and CreateDisposition.CREATE_NEVER while the target table does not exist (or is not visible to the caller), so SCHEMA_CACHE returns null and no schema can be resolved.

Common situations: Typo in dataset/table name; table created in a different project than configured; running with credentials/permissions that cannot see the table; assuming CREATE_NEVER auto-creates tables (it never does); renamed or dropped tables between pipeline setup and execution.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Table not found
• Table not found:
• Failed to patch table schema.
• Failed to flush elements on window expiration!
• Schema must be set for table " + tableReference + " when wri

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b176873bf12b75e1. Report an issue: GitHub.