{"record":{"id":"dfefed18dda20da7","repo":"dbt-labs/dbt-core","slug":"jsonify-nested-columns-rewritten-schema-and-colum","errorCode":null,"errorMessage":"jsonify_nested_columns: rewritten schema and columns are consistent","messagePattern":"jsonify_nested_columns: rewritten schema and columns are consistent","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/dbt-adapter/src/record_batch.rs","lineNumber":219,"sourceCode":"                let (encode_field, encode_column) = jsonify_map_keys(field, column, &options);\n                let string_col = encode_array_to_strings(&encode_field, &encode_column, &options);\n                new_columns.push(Arc::new(string_col));\n                new_fields.push(Arc::new(\n                    Field::new(field.name(), DataType::Utf8, field.is_nullable())\n                        .with_metadata(field.metadata().clone()),\n                ));\n            } else {\n                new_columns.push(column.clone());\n                new_fields.push(field.clone());\n            }\n        }\n\n        let new_schema = Arc::new(Schema::new_with_metadata(\n            new_fields,\n            schema.metadata().clone(),\n        ));\n        RecordBatch::try_new(new_schema, new_columns)\n            .expect(\"jsonify_nested_columns: rewritten schema and columns are consistent\")\n    }\n\n    fn lowercase_column_names(self) -> RecordBatch {\n        let schema = self.schema();\n        let fields = schema.fields();\n\n        if fields.iter().all(|f| {\n            f.name()\n                .chars()\n                .all(|c| c.is_lowercase() || !c.is_alphabetic())\n        }) {\n            return self;\n        }\n\n        let new_fields: Vec<_> = fields\n            .iter()\n            .map(|f| Arc::new(f.as_ref().clone().with_name(f.name().to_lowercase())))\n            .collect();","sourceCodeStart":201,"sourceCodeEnd":237,"githubUrl":"https://github.com/dbt-labs/dbt-core/blob/0267ce9170576975b76b64ce856b2e5848e96617/crates/dbt-adapter/src/record_batch.rs#L201-L237","documentation":"This panic fires when rebuilding a RecordBatch after converting nested (struct/list/map) columns to JSON strings. The code builds new_fields and new_columns in lockstep, so arrow's RecordBatch::try_new should always accept them; if it returns Err, the length or datatype alignment between the rewritten schema and columns is broken, which the library treats as an unrecoverable internal invariant violation and panics via .expect.","triggerScenarios":"Calling jsonify_nested_columns() on a RecordBatch where the jsonify rewrite produced a column whose length differs from the new schema's field count, or where a field's DataType was changed without correspondingly rebuilding the column array (e.g. a nested column not actually replaced by a Utf8 JSON string).","commonSituations":"Custom arrow-rs version mismatches where arrow_json encoder output types differ; driver-specific column widening code that runs before jsonify and leaves schema/arrays inconsistent; downstream forks adding new nested DataType handling without updating both new_fields and new_columns.","solutions":["Check that every field added to new_fields has a matching column in new_columns with the same length and DataType (Utf8 for JSONified nested columns).","Update the arrow-rs / arrow-json dependency to the version pinned by the workspace so encoder behavior matches assumptions.","File a bug with the offending schema (print new_fields and column types) since this indicates a bug in the adapter, not user input.","As a temporary workaround, bypass jsonify_nested_columns and cast nested columns manually with cast_with_options before consuming the batch."],"exampleFix":"// before\nRecordBatch::try_new(new_schema, new_columns)\n    .expect(\"jsonify_nested_columns: rewritten schema and columns are consistent\")\n// after\nRecordBatch::try_new(new_schema, new_columns).map_err(|e| {\n    AdapterError::new(AdapterErrorKind::Internal, format!(\n        \"jsonify_nested_columns produced inconsistent batch: {e}\"))\n})?","handlingStrategy":"validation","validationCode":"fn batch_consistent(fields: &[FieldRef], cols: &[ArrayRef]) -> bool {\n    fields.len() == cols.len()\n        && fields.iter().zip(cols).all(|(f, c)| c.len() == f.len().max(cols[0].len()) && c.data_type() == f.data_type())\n}\n// call before jsonify_nested_columns-dependent processing","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep arrow-rs/arrow-json versions aligned with the adapter's lockfile","After any schema rewrite, immediately assert fields.len() == columns.len() and equal array lengths","Never mutate RecordBatch columns without rebuilding the schema in the same step","Add a debug_assert in custom drivers that built batches satisfy RecordBatch::try_new"],"tags":["arrow","record-batch","panic","internal-invariant"],"backgroundTag":"internal-invariant-violation","analyzedSha":"0267ce9170576975b76b64ce856b2e5848e96617","analyzedAt":"2026-09-07T21:53:39.732Z","contentChangedAt":"2026-09-07T21:53:39.732Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}