dbt-labs/dbt-core · error

column name normalization preserves schema compatibility

Error message

column name normalization preserves schema compatibility

What it means

`lowercase_column_names` (crates/dbt-adapter/src/metadata/snowflake/mod.rs:406) rebuilds a `RecordBatch` with lowercased field names via `RecordBatch::try_new(...).expect(...)`. Arrow guarantees `try_new` succeeds when the new schema has the same field count, types, and nullability as the original columns; the panic fires only if a column's type/nullability no longer matches the schema.

Source

Thrown at crates/dbt-adapter/src/metadata/snowflake/mod.rs:406

pub const ARROW_FIELD_SNOWFLAKE_FIELD_WIDTH_METADATA_KEY: &str = "SNOWFLAKE:field_width";

/// Normalize all column names in a RecordBatch to lowercase.
///
/// Snowflake may uppercase column aliases (e.g. `table_catalog as "table_database"`) depending
/// on account-level settings, even when the alias is double-quoted. Lowercasing the schema up
/// front lets all downstream `get_column_values` calls use their expected lowercase names without
/// needing per-call case-insensitive logic.
fn lowercase_column_names(batch: &RecordBatch) -> RecordBatch {
    let schema = batch.schema();
    let fields: Vec<_> = schema
        .fields()
        .iter()
        .map(|f| Arc::new(f.as_ref().clone().with_name(f.name().to_lowercase())))
        .collect();
    let new_schema = Arc::new(Schema::new_with_metadata(fields, schema.metadata().clone()));
    RecordBatch::try_new(new_schema, batch.columns().to_vec())
        .expect("column name normalization preserves schema compatibility")
}

fn accumulate_view_definition_fetch_result(
    acc: &mut ViewDefinitionFetchResult,
    batch: &RecordBatch,
) -> AdapterResult<()> {
    let batch = lowercase_column_names(batch);
    // Result schema: (fqn STRING, view_definition STRING, error STRING)
    let fqns_arr = batch.column_values::<StringArray>("fqn")?;
    let defs_arr = batch.column_values::<StringArray>("view_definition")?;

    for i in 0..batch.num_rows() {
        let fqn = fqns_arr.value(i).to_string();
        if defs_arr.is_null(i) {
            // A NULL definition means Snowflake could not return DDL for
            // this relation. Treat it as unresolvable regardless of the
            // exact GET_DDL error text so the query cache can fall back
            // to freshness metadata for secure/data-share views.

View on GitHub (pinned to 0267ce9170)

Solutions

1. Verify the source batch was constructed with a schema matching its columns; fix the producer
2. Propagate the `try_new` error instead of expecting, so the offending schema is reported
3. Ensure `with_name` preserves all field attributes (nullability, metadata, extensions) when cloning fields
4. Pin compatible arrow-rs versions across the workspace

Example fix

// before
.expect("column name normalization preserves schema compatibility")
// after
.map_err(|e| AdapterError::new(AdapterErrorKind::UnexpectedResult, e.to_string()))?

// verify batch schema matches columns before lowering names
assert_eq!(schema.fields().len(), batch.num_columns());
assert!(schema.fields().iter().zip(batch.columns()).all(|(f, c)| f.data_type() == c.data_type()));

fn batch_matches_schema(batch: &RecordBatch) -> bool {
    batch.schema().fields().len() == batch.num_columns()
        && batch.schema().fields().iter().zip(batch.columns()).all(|(f, c)| f.data_type() == c.data_type())
}

match RecordBatch::try_new(new_schema, cols) { Ok(b) => b, Err(e) => return Err(...) }

When it happens

Trigger: Lowercasing columns of a batch whose fields have mismatched Arrow data types or nullability flags relative to the actual column arrays (e.g. a producer built the batch with inconsistent schema/columns).

Common situations: Upstream Arrow version changes altering nullability defaults; adapter code building batches with schema/column type drift; dictionary or view-typed columns whose rebuilt schema loses an attribute.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Schema not found for canonical FQN: {}
• expected_field should be Struct
• {e}
• Not all tables have the same column types!
• Not all tables have the same column names!

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/8dd81be15ee83317. Report an issue: GitHub.