{"record":{"id":"0a69243f8c8d540c","repo":"apache/seatunnel","slug":"target-bigquery-table-s-is-missing-source-colum","errorCode":null,"errorMessage":"Target BigQuery table '%s' is missing source column: '%s'","messagePattern":"Target BigQuery table '(.+?)' is missing source column: '(.+?)'","errorType":"exception","errorClass":"CatalogException","httpStatus":null,"severity":"error","filePath":"seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySaveModeHandler.java","lineNumber":105,"sourceCode":"        if (schemaSaveMode == SchemaSaveMode.RECREATE_SCHEMA\n                || schemaSaveMode == SchemaSaveMode.IGNORE) {\n            return;\n        }\n\n        try {\n            CatalogTable remoteTable = catalog.getTable(tablePath);\n            if (remoteTable == null || remoteTable.getTableSchema() == null) {\n                return;\n            }\n\n            for (Column sourceColumn : catalogTable.getTableSchema().getColumns()) {\n                Optional<Column> remoteColOpt =\n                        remoteTable.getTableSchema().getColumns().stream()\n                                .filter(c -> c.getName().equalsIgnoreCase(sourceColumn.getName()))\n                                .findFirst();\n\n                if (!remoteColOpt.isPresent()) {\n                    throw new CatalogException(\n                            String.format(\n                                    \"Target BigQuery table '%s' is missing source column: '%s'\",\n                                    tablePath.getFullName(), sourceColumn.getName()));\n                }\n\n                Column remoteColumn = remoteColOpt.get();\n                if (!isTypeCompatible(\n                        sourceColumn.getDataType().getSqlType(),\n                        remoteColumn.getDataType().getSqlType())) {\n                    throw new CatalogException(\n                            String.format(\n                                    \"Type mismatch for column '%s' inside target BigQuery table '%s'. Source type: '%s', target type: '%s'\",\n                                    sourceColumn.getName(),\n                                    tablePath.getFullName(),\n                                    sourceColumn.getDataType().getSqlType(),\n                                    remoteColumn.getDataType().getSqlType()));\n                }\n            }","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySaveModeHandler.java#L87-L123","documentation":"This CatalogException is thrown by BigQuerySaveModeHandler.validateSchemaCoherence during handleSchemaSaveMode when a column present in the SeaTunnel source schema is not found (case-insensitive name match) in the remote BigQuery table schema. It fires in save-mode 'error' handling where the sink schema must fully contain the source columns. The job fails fast rather than writing mismatched data.","triggerScenarios":"Running a sink with schema_save_mode=ERROR (default) when the target BigQuery table exists but lacks a column the source produces — e.g. source schema evolved to add a field, the remote table was created from an older version, or the column name casing differs in a way the equality filter does not match (leading/trailing spaces, special chars).","commonSituations":"Upstream added a column and the BigQuery table was not ALTERed; misconfigured table name pointing to a different table; auto-create disabled so an old table is targeted; column renamed in the source config.","solutions":["ALTER TABLE the BigQuery table to add the missing column, or recreate the table to match the source schema","Set schema_save_mode=CREATE_SCHEMA_WHEN_NOT_EXIST or EVOLVE_SCHEMA so the sink syncs the schema automatically","Verify the table_path points at the intended BigQuery table","Align source column names with the BigQuery table (rename in the transform/config if needed)"],"exampleFix":"// config before\nschema_save_mode = ERROR\n// config after\nschema_save_mode = EVOLVE_SCHEMA  // or ALTER TABLE ... ADD COLUMN on the BigQuery table","handlingStrategy":"validation","validationCode":"TableSchema source = ...; Table remote = catalog.getTable(tablePath);\nif (remote != null) {\n    Set<String> remoteCols = remote.getTableSchema().getColumnNames().stream()\n        .map(String::toUpperCase).collect(Collectors.toSet());\n    List<String> missing = source.getColumns().stream()\n        .map(Column::getName)\n        .filter(n -> !remoteCols.contains(n.toUpperCase()))\n        .collect(Collectors.toList());\n    if (!missing.isEmpty()) {\n        LOG.error(\"Columns missing in BigQuery table \" + tablePath.getFullName() + \": \" + missing);\n    }\n}","typeGuard":null,"tryCatchPattern":"try {\n    handler.handleSchemaSaveMode();\n} catch (CatalogException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"missing source column\")) {\n        LOG.error(\"Sync BigQuery table schema with source, or set schema_save_mode=EVOLVE_SCHEMA\");\n    }\n    throw e;\n}","preventionTips":["Run schema drift checks between source and BigQuery table before jobs","Use schema_save_mode=EVOLVE_SCHEMA or CREATE_SCHEMA_WHEN_NOT_EXIST for evolving pipelines","Ensure ALTER TABLE ADD COLUMN is applied when upstream adds fields","Verify table_path targets the intended table"],"tags":["bigquery","schema-validation","save-mode","column-missing"],"backgroundTag":"schema-validation-failed","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}