What causes "Target BigQuery table '%s' is missing source column: '%s'" in apache/seatunnel?

ALTER TABLE the BigQuery table to add the missing column, or recreate the table to match the source schema

apache/seatunnel · error · CatalogException

Target BigQuery table

Error message

Target BigQuery table '%s' is missing source column: '%s'

What it means

This CatalogException is thrown by BigQuerySaveModeHandler.validateSchemaCoherence during handleSchemaSaveMode when a column present in the SeaTunnel source schema is not found (case-insensitive name match) in the remote BigQuery table schema. It fires in save-mode 'error' handling where the sink schema must fully contain the source columns. The job fails fast rather than writing mismatched data.

Solutions

ALTER TABLE the BigQuery table to add the missing column, or recreate the table to match the source schema
Set schema_save_mode=CREATE_SCHEMA_WHEN_NOT_EXIST or EVOLVE_SCHEMA so the sink syncs the schema automatically
Verify the table_path points at the intended BigQuery table
Align source column names with the BigQuery table (rename in the transform/config if needed)

Example fix

// config before
schema_save_mode = ERROR
// config after
schema_save_mode = EVOLVE_SCHEMA  // or ALTER TABLE ... ADD COLUMN on the BigQuery table

Defensive patterns

Strategy: validation

Validate before calling

TableSchema source = ...; Table remote = catalog.getTable(tablePath);
if (remote != null) {
    Set<String> remoteCols = remote.getTableSchema().getColumnNames().stream()
        .map(String::toUpperCase).collect(Collectors.toSet());
    List<String> missing = source.getColumns().stream()
        .map(Column::getName)
        .filter(n -> !remoteCols.contains(n.toUpperCase()))
        .collect(Collectors.toList());
    if (!missing.isEmpty()) {
        LOG.error("Columns missing in BigQuery table " + tablePath.getFullName() + ": " + missing);
    }
}

Try / catch

try {
    handler.handleSchemaSaveMode();
} catch (CatalogException e) {
    if (e.getMessage() != null && e.getMessage().contains("missing source column")) {
        LOG.error("Sync BigQuery table schema with source, or set schema_save_mode=EVOLVE_SCHEMA");
    }
    throw e;
}

Prevention

Run schema drift checks between source and BigQuery table before jobs
Use schema_save_mode=EVOLVE_SCHEMA or CREATE_SCHEMA_WHEN_NOT_EXIST for evolving pipelines
Ensure ALTER TABLE ADD COLUMN is applied when upstream adds fields
Verify table_path targets the intended table

When it happens

Trigger: Running a sink with schema_save_mode=ERROR (default) when the target BigQuery table exists but lacks a column the source produces — e.g. source schema evolved to add a field, the remote table was created from an older version, or the column name casing differs in a way the equality filter does not match (leading/trailing spaces, special chars).

Common situations: Upstream added a column and the BigQuery table was not ALTERed; misconfigured table name pointing to a different table; auto-create disabled so an old table is targeted; column renamed in the source config.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/0a69243f8c8d540c. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySaveModeHandler.java:105

        if (schemaSaveMode == SchemaSaveMode.RECREATE_SCHEMA
                || schemaSaveMode == SchemaSaveMode.IGNORE) {
            return;
        }

        try {
            CatalogTable remoteTable = catalog.getTable(tablePath);
            if (remoteTable == null || remoteTable.getTableSchema() == null) {
                return;
            }

            for (Column sourceColumn : catalogTable.getTableSchema().getColumns()) {
                Optional<Column> remoteColOpt =
                        remoteTable.getTableSchema().getColumns().stream()
                                .filter(c -> c.getName().equalsIgnoreCase(sourceColumn.getName()))
                                .findFirst();

                if (!remoteColOpt.isPresent()) {
                    throw new CatalogException(
                            String.format(
                                    "Target BigQuery table '%s' is missing source column: '%s'",
                                    tablePath.getFullName(), sourceColumn.getName()));
                }

                Column remoteColumn = remoteColOpt.get();
                if (!isTypeCompatible(
                        sourceColumn.getDataType().getSqlType(),
                        remoteColumn.getDataType().getSqlType())) {
                    throw new CatalogException(
                            String.format(
                                    "Type mismatch for column '%s' inside target BigQuery table '%s'. Source type: '%s', target type: '%s'",
                                    sourceColumn.getName(),
                                    tablePath.getFullName(),
                                    sourceColumn.getDataType().getSqlType(),
                                    remoteColumn.getDataType().getSqlType()));
                }
            }

View on GitHub (pinned to cf67b549a7)