dbt-labs/dbt-core · error

make_encoder for nested column should not fail

Error message

make_encoder for nested column should not fail

What it means

encode_array_to_strings uses arrow_json's make_encoder to serialize a nested array (list/struct/map) into JSON strings. make_encoder returns Result and can reject unsupported DataTypes; this code asserts that for nested columns produced by jsonify_nested_columns/jsonify_map_keys the encoder is always constructible. If it returns Err, the array's DataType is not encodable — an internal invariant violation, surfaced as a panic.

Source

Thrown at crates/dbt-adapter/src/record_batch.rs:290

impl SchemaExt for Schema {
    fn has_dml_columns(&self, adapter_type: AdapterType) -> bool {
        match adapter_type {
            AdapterType::Snowflake => self
                .fields()
                .iter()
                .any(|f| SNOWFLAKE_DML_COLUMNS.contains(&f.name().as_str())),
            _ => false,
        }
    }
}

fn encode_array_to_strings(
    field: &FieldRef,
    array: &ArrayRef,
    options: &EncoderOptions,
) -> StringArray {
    let mut encoder = make_encoder(field, array.as_ref(), options)
        .expect("make_encoder for nested column should not fail");
    let mut builder = StringBuilder::with_capacity(array.len(), array.len() * 32);
    let mut buf: Vec<u8> = Vec::with_capacity(64);
    for row in 0..array.len() {
        if encoder.is_null(row) {
            builder.append_null();
        } else {
            buf.clear();
            encoder.encode(row, &mut buf);
            let s = std::str::from_utf8(&buf).expect("arrow_json::Encoder emits UTF-8");
            builder.append_value(s);
        }
    }
    builder.finish()
}

/// Recursively rewrite every nested `Map` so its keys become `Utf8`.
///
/// arrow_json's map encoder only supports UTF-8 keys, while dbt Core stringifies any key via

View on GitHub (pinned to 0267ce9170)

Solutions

1. Print/inspect the field's DataType where it panics and confirm it is a supported nested type (Struct, List, LargeList, FixedSizeList, Map).
2. Upgrade or pin arrow-json to the workspace-pinned version so encoder support matches the DataTypes the adapter emits.
3. Extend the dispatch in jsonify_nested_columns to route unsupported DataTypes through a fallback (e.g. cast to string) instead of make_encoder.
4. Report the DataType to the adapter maintainers — this path is expected to always succeed.

Example fix

// before
let mut encoder = make_encoder(field, array.as_ref(), options)
    .expect("make_encoder for nested column should not fail");
// after
let mut encoder = make_encoder(field, array.as_ref(), options).map_err(|e| {
    AdapterError::new(AdapterErrorKind::Internal, format!(
        "make_encoder failed for {}: {e}", field.data_type()))
})?;

fn is_json_encodable(dt: &DataType) -> bool {
    matches!(dt, DataType::Struct(_) | DataType::List(_) | DataType::LargeList(_) | DataType::FixedSizeList(_, _) | DataType::Map(_, _))
}
// route non-nested/unsupported types away from encode_array_to_strings

fn guard_nested(field: &FieldRef) -> bool {
    matches!(field.data_type(), DataType::Struct(_) | DataType::List(_) | DataType::LargeList(_) | DataType::FixedSizeList(_, _) | DataType::Map(_, _))
}

When it happens

Trigger: Calling encode_array_to_strings (via jsonify_nested_columns or jsonify_map_keys) on an array whose DataType arrow_json's make_encoder does not support (e.g. an exotic or newly added arrow type, or a non-nested type reaching the function by mistake).

Common situations: Upgrading arrow-rs introduces/renames DataTypes the encoder doesn't handle; a driver returns an unusual nested variant (e.g. large-list of views) not covered by the encoder; misrouting a plain column into the nested-encoding path.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• arrow_json::Encoder emits UTF-8
• expected_field should be Struct
• jsonify_nested_columns: rewritten schema and columns are con
• lowercase_column_names: schema and columns should be compati
• string prefix checked

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/d27a426a93981413. Report an issue: GitHub.