apache/seatunnel · error · CatalogException

Failed to create BigQuery table:

Error message

Failed to create BigQuery table: 

What it means

BigQueryCatalog.createTable wraps exceptions from the actual bigquery.create(TableInfo.of(tableId, tableDefinition)) API call in a CatalogException naming the fully-qualified table. The schema passed local validation (including the streaming-PK check), so this failure comes from the BigQuery service: HTTP-level rejection of the create request. The real cause from the google-cloud-bigquery client is attached.

Source

Thrown at seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/catalog/BigQueryCatalog.java:296

            com.google.cloud.bigquery.TableConstraints constraints =
                    com.google.cloud.bigquery.TableConstraints.newBuilder()
                            .setPrimaryKey(bqPrimaryKey)
                            .build();

            tableDefinition =
                    StandardTableDefinition.newBuilder()
                            .setSchema(bqSchema)
                            .setTableConstraints(constraints)
                            .build();
        } else {
            tableDefinition = StandardTableDefinition.of(bqSchema);
        }

        try {
            bigquery.create(TableInfo.of(tableId, tableDefinition));
            log.info("BigQuery Table '{}' created successfully.", tablePath.getFullName());
        } catch (Exception e) {
            throw new CatalogException(
                    "Failed to create BigQuery table: " + tablePath.getFullName(), e);
        }
    }

    @Override
    public void dropTable(TablePath tablePath, boolean ignoreIfNotExists)
            throws TableNotExistException, CatalogException {
        TableId tableId = TableId.of(getDatasetName(tablePath), tablePath.getTableName());
        try {
            boolean deleted = bigquery.delete(tableId);
            if (!deleted && !ignoreIfNotExists) {
                throw new TableNotExistException(catalogName, tablePath);
            }
            log.info("BigQuery Table '{}' dropped successfully.", tablePath.getFullName());
        } catch (TableNotExistException e) {
            throw e;
        } catch (Exception e) {
            throw new CatalogException(

View on GitHub (pinned to cf67b549a7)

Solutions

1. Read the wrapped cause: 409 'Already Exists' -> enable CREATE_SCHEMA_WHEN_NOT_EXISTS-style handling or call createTable only after tableExists check with ignoreIfNotExists semantics.
2. Create the dataset first: `bq mk --dataset PROJECT:dataset` or ensure the catalog config allows database creation (create_database_if_not_exists = true).
3. Grant `roles/bigquery.dataEditor` (tables.create permission) to the service account on the dataset.
4. Check the generated bqSchema: log the SeaTunnel CatalogTable columns and verify each type maps to a supported BigQuery field type; rename columns with illegal characters.
5. If a type is unmappable, add a transform to cast it to a supported type before the sink.

Example fix

// before: creating table without its dataset
// dataset "analytics" does not exist -> Failed to create BigQuery table: project.analytics.orders

// after: create dataset first (or enable auto-create)
# bq mk --dataset myproject:analytics
// or in catalog options:
// create_database_if_not_exists = true

// Java, pre-flight checks before createTable
Dataset dataset = bigquery.getDataset(DatasetId.of(project, datasetName));
if (dataset == null) {
    bigquery.create(DatasetInfo.of(project, datasetName)); // or fail with clear message
}
if (bigquery.getTable(TableId.of(project, datasetName, tableName)) != null) {
    // decide: skip, or throw with ignoreIfExists semantics
    log.info("Table already exists: {}.{}.{}", project, datasetName, tableName);
}
// validate all columns map to supported BigQuery types
schema.getColumns().forEach(c ->
    checkArgument(SUPPORTED_TYPES.contains(c.getSourceType()),
        "Unsupported type for BigQuery: " + c.getSourceType()));

static boolean canCreateTable(BigQuery bq, String project, String dataset) {
    Dataset d = bq.getDataset(DatasetId.of(project, dataset));
    return d != null; // caller must additionally hold tables.create IAM permission
}

try {
    catalog.createTable(tablePath, table, ignoreIfExists);
} catch (CatalogException e) {
    if (e.getCause() instanceof BigQueryException be) {
        if (be.getCode() == 409) {
            log.info("Table {} already exists; skipping", tablePath.getFullName());
            return;
        }
        if (be.getCode() == 404) {
            throw new IllegalStateException("Dataset missing for " + tablePath.getFullName(), e);
        }
    }
    throw e;
}

When it happens

Trigger: bigquery.create() fails: table already exists when ignore rules don't apply, invalid field names/types in the generated BigQuery Schema (unsupported SeaTunnel type mapping, names with illegal chars), caller lacks bigquery.tables.create on the dataset, dataset does not exist, exceeding table limits, or network/quota errors.

Common situations: schema_save_mode = CREATE always and the table already exists; dataset was never created (create_database_if_not_exists not enabled); SeaTunnel types that map to unsupported BigQuery types (e.g. complex/map/array mappings) in the converted Schema; names containing characters BigQuery rejects; IAM changes removing the SA's dataEditor role.

Related errors

• Failed to list tables in dataset:
• Failed to drop BigQuery table:
• Failed to open BigQueryCatalog
• Failed to list BigQuery databases (datasets)
• Streaming mode requires a Primary Key in the schema of table

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/04082fa0b021626f. Report an issue: GitHub.