dbt-labs/dbt-core · error

arrow_json::Encoder emits UTF-8

Error message

arrow_json::Encoder emits UTF-8

What it means

After arrow_json's Encoder writes a row into a byte buffer, the code converts those bytes to &str asserting the encoder only emits valid UTF-8. arrow_json's contract is to produce JSON text (UTF-8 by definition); a failure means the buffer contains non-UTF-8 bytes, which the library treats as an encoder-contract violation and panics via .expect.

Source

Thrown at crates/dbt-adapter/src/record_batch.rs:299

    }
}

fn encode_array_to_strings(
    field: &FieldRef,
    array: &ArrayRef,
    options: &EncoderOptions,
) -> StringArray {
    let mut encoder = make_encoder(field, array.as_ref(), options)
        .expect("make_encoder for nested column should not fail");
    let mut builder = StringBuilder::with_capacity(array.len(), array.len() * 32);
    let mut buf: Vec<u8> = Vec::with_capacity(64);
    for row in 0..array.len() {
        if encoder.is_null(row) {
            builder.append_null();
        } else {
            buf.clear();
            encoder.encode(row, &mut buf);
            let s = std::str::from_utf8(&buf).expect("arrow_json::Encoder emits UTF-8");
            builder.append_value(s);
        }
    }
    builder.finish()
}

/// Recursively rewrite every nested `Map` so its keys become `Utf8`.
///
/// arrow_json's map encoder only supports UTF-8 keys, while dbt Core stringifies any key via
/// `json.dumps`. Non-string keys (integers, floats, structs, lists, ...) are JSON-encoded into
/// their string form so the resulting map serializes to valid JSON.
fn jsonify_map_keys(
    field: &FieldRef,
    array: &ArrayRef,
    options: &EncoderOptions,
) -> (FieldRef, ArrayRef) {
    match field.data_type() {
        DataType::Map(entries, ordered) => {

View on GitHub (pinned to 0267ce9170)

Solutions

1. Verify the arrow-json version matches the workspace pin and restore the stock encoder if it was patched.
2. Inspect the failing row's data: non-UTF-8 strings in nested columns should be sanitized by the driver before reaching the encoder.
3. Replace .expect with a lossy fallback (String::from_utf8_lossy) plus a warning if you must keep processing.
4. Report the offending data/row to maintainers; upstream arrow_json guarantees UTF-8 output.

Example fix

// before
let s = std::str::from_utf8(&buf).expect("arrow_json::Encoder emits UTF-8");
// after
let s = std::str::from_utf8(&buf).map_err(|e| {
    AdapterError::new(AdapterErrorKind::Internal, format!(
        "encoder emitted non-UTF-8 bytes: {e}"))
})?;

// Pre-check is not possible on the internal buffer; validate output instead:
fn is_utf8(buf: &[u8]) -> bool { std::str::from_utf8(buf).is_ok() }

match std::str::from_utf8(&buf) {
    Ok(s) => builder.append_value(s),
    Err(_) => builder.append_value(String::from_utf8_lossy(&buf)), // or log & fail loudly
}

When it happens

Trigger: Calling encode_array_to_strings (via jsonify_nested_columns or jsonify_map_keys) where encoder.encode() writes bytes that std::str::from_utf8 rejects — e.g. a custom/patched encoder emitting raw non-UTF-8 payloads or a buffer mixing content from a non-UTF-8 source.

Common situations: Forked or outdated arrow-json versions whose encoder behavior deviates (e.g. emitting escaped latin-1 bytes); injecting a custom EncoderOptions/encoder implementation; memory corruption is essentially ruled out in safe Rust.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• make_encoder for nested column should not fail
• expected_field should be Struct
• jsonify_nested_columns: rewritten schema and columns are con
• lowercase_column_names: schema and columns should be compati
• string prefix checked

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/fdd82289527ca101. Report an issue: GitHub.