dbt-labs/dbt-core · error

jsonify_nested_columns: rewritten schema and columns are…

Error message

jsonify_nested_columns: rewritten schema and columns are consistent

What it means

This panic fires when rebuilding a RecordBatch after converting nested (struct/list/map) columns to JSON strings. The code builds new_fields and new_columns in lockstep, so arrow's RecordBatch::try_new should always accept them; if it returns Err, the length or datatype alignment between the rewritten schema and columns is broken, which the library treats as an unrecoverable internal invariant violation and panics via .expect.

Solutions

1. Check that every field added to new_fields has a matching column in new_columns with the same length and DataType (Utf8 for JSONified nested columns).
2. Update the arrow-rs / arrow-json dependency to the version pinned by the workspace so encoder behavior matches assumptions.
3. File a bug with the offending schema (print new_fields and column types) since this indicates a bug in the adapter, not user input.
4. As a temporary workaround, bypass jsonify_nested_columns and cast nested columns manually with cast_with_options before consuming the batch.

Example fix

// before
RecordBatch::try_new(new_schema, new_columns)
    .expect("jsonify_nested_columns: rewritten schema and columns are consistent")
// after
RecordBatch::try_new(new_schema, new_columns).map_err(|e| {
    AdapterError::new(AdapterErrorKind::Internal, format!(
        "jsonify_nested_columns produced inconsistent batch: {e}"))
})?

fn batch_consistent(fields: &[FieldRef], cols: &[ArrayRef]) -> bool {
    fields.len() == cols.len()
        && fields.iter().zip(cols).all(|(f, c)| c.len() == f.len().max(cols[0].len()) && c.data_type() == f.data_type())
}
// call before jsonify_nested_columns-dependent processing

When it happens

Trigger: Calling jsonify_nested_columns() on a RecordBatch where the jsonify rewrite produced a column whose length differs from the new schema's field count, or where a field's DataType was changed without correspondingly rebuilding the column array (e.g. a nested column not actually replaced by a Utf8 JSON string).

Common situations: Custom arrow-rs version mismatches where arrow_json encoder output types differ; driver-specific column widening code that runs before jsonify and leaves schema/arrays inconsistent; downstream forks adding new nested DataType handling without updating both new_fields and new_columns.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• lowercase_column_names: schema and columns should be…
• string prefix checked
• AgateTable exists
• arrow_json::Encoder emits UTF-8
• expected_field should be Struct

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/dfefed18dda20da7. Report an issue: GitHub.

Appendix: source

Thrown at crates/dbt-adapter/src/record_batch.rs:219

                let (encode_field, encode_column) = jsonify_map_keys(field, column, &options);
                let string_col = encode_array_to_strings(&encode_field, &encode_column, &options);
                new_columns.push(Arc::new(string_col));
                new_fields.push(Arc::new(
                    Field::new(field.name(), DataType::Utf8, field.is_nullable())
                        .with_metadata(field.metadata().clone()),
                ));
            } else {
                new_columns.push(column.clone());
                new_fields.push(field.clone());
            }
        }

        let new_schema = Arc::new(Schema::new_with_metadata(
            new_fields,
            schema.metadata().clone(),
        ));
        RecordBatch::try_new(new_schema, new_columns)
            .expect("jsonify_nested_columns: rewritten schema and columns are consistent")
    }

    fn lowercase_column_names(self) -> RecordBatch {
        let schema = self.schema();
        let fields = schema.fields();

        if fields.iter().all(|f| {
            f.name()
                .chars()
                .all(|c| c.is_lowercase() || !c.is_alphabetic())
        }) {
            return self;
        }

        let new_fields: Vec<_> = fields
            .iter()
            .map(|f| Arc::new(f.as_ref().clone().with_name(f.name().to_lowercase())))
            .collect();

View on GitHub (pinned to 0267ce9170)