apache/seatunnel · error · CatalogException

Failed to validate BigQuery schema coherence

Error message

Failed to validate BigQuery schema coherence

What it means

Thrown by BigQuerySaveModeHandler.validateSchemaCoherence when the schema validation itself throws a BigQueryException (e.g. failure fetching the remote table schema via BigQuery API) that is neither CatalogException nor SeaTunnelRuntimeException. It wraps the API failure while preserving the cause. Notably, other unexpected exceptions are only logged as a warning and skipped — only BigQueryException fails the job with this message.

Solutions

1. Inspect the wrapped BigQueryException cause for the specific API error (403, 404, 429, 5xx)
2. Verify the service account can read table metadata (bigquery.tables.get, metadataViewer)
3. Retry the job if the cause was a transient quota/network error
4. Confirm table_path resolves to an existing table before save-mode validation

if (!catalog.tableExists(tablePath)) {
    throw new IllegalStateException("BigQuery table " + tablePath.getFullName() + " missing before schema validation");
}

try {
    handler.handleSchemaSaveMode();
} catch (CatalogException e) {
    if (e.getCause() instanceof BigQueryException bqe && bqe.getCode() == 429) {
        // transient quota error: backoff and retry the job/validation
    } else {
        throw e;
    }
}

When it happens

Trigger: During handleSchemaSaveMode, when retrieving the remote BigQuery table schema (get remote table) fails due to a BigQueryException: table lookup rejected, permission denied on metadata, quota/network errors, or invalid dataset/table identifiers.

Common situations: Transient BigQuery API outage or rate limit while validating schema at job start; service account lacking bigquery.tables.get; table deleted between existence check and schema read; malformed table path.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Target BigQuery table
• Type mismatch for column
• Airtable API request failed, status code
• All candidate sink tables were skipped in Spark starter.
• API-09

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/52d6400faec0c748. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySaveModeHandler.java:140

                }
            }
            log.info(
                    "BigQuery schema coherence check passed successfully for table: {}",
                    tablePath.getFullName());
        } catch (TableNotExistException e) {
            log.info(
                    "Target BigQuery table '{}' does not exist yet. Skipping schema coherence check.",
                    tablePath.getFullName());
            return;
        } catch (Exception e) {
            if (e instanceof CatalogException) {
                throw (CatalogException) e;
            }
            if (e instanceof SeaTunnelRuntimeException) {
                throw (SeaTunnelRuntimeException) e;
            }
            if (e instanceof BigQueryException) {
                throw new CatalogException("Failed to validate BigQuery schema coherence", e);
            }
            log.warn("Schema validation check ignored due to exception: {}", e.getMessage());
        }
    }

    private boolean isTypeCompatible(SqlType source, SqlType sink) {
        if (source == sink) {
            return true;
        }
        // Widening integer compatibility
        if ((source == SqlType.TINYINT
                        || source == SqlType.SMALLINT
                        || source == SqlType.INT
                        || source == SqlType.BIGINT)
                && (sink == SqlType.BIGINT)) {
            return true;
        }
        // Widening float compatibility

View on GitHub (pinned to cf67b549a7)