{"record":{"id":"fdd82289527ca101","repo":"dbt-labs/dbt-core","slug":"arrow-json-encoder-emits-utf-8","errorCode":null,"errorMessage":"arrow_json::Encoder emits UTF-8","messagePattern":"arrow_json::Encoder emits UTF-8","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/dbt-adapter/src/record_batch.rs","lineNumber":299,"sourceCode":"    }\n}\n\nfn encode_array_to_strings(\n    field: &FieldRef,\n    array: &ArrayRef,\n    options: &EncoderOptions,\n) -> StringArray {\n    let mut encoder = make_encoder(field, array.as_ref(), options)\n        .expect(\"make_encoder for nested column should not fail\");\n    let mut builder = StringBuilder::with_capacity(array.len(), array.len() * 32);\n    let mut buf: Vec<u8> = Vec::with_capacity(64);\n    for row in 0..array.len() {\n        if encoder.is_null(row) {\n            builder.append_null();\n        } else {\n            buf.clear();\n            encoder.encode(row, &mut buf);\n            let s = std::str::from_utf8(&buf).expect(\"arrow_json::Encoder emits UTF-8\");\n            builder.append_value(s);\n        }\n    }\n    builder.finish()\n}\n\n/// Recursively rewrite every nested `Map` so its keys become `Utf8`.\n///\n/// arrow_json's map encoder only supports UTF-8 keys, while dbt Core stringifies any key via\n/// `json.dumps`. Non-string keys (integers, floats, structs, lists, ...) are JSON-encoded into\n/// their string form so the resulting map serializes to valid JSON.\nfn jsonify_map_keys(\n    field: &FieldRef,\n    array: &ArrayRef,\n    options: &EncoderOptions,\n) -> (FieldRef, ArrayRef) {\n    match field.data_type() {\n        DataType::Map(entries, ordered) => {","sourceCodeStart":281,"sourceCodeEnd":317,"githubUrl":"https://github.com/dbt-labs/dbt-core/blob/0267ce9170576975b76b64ce856b2e5848e96617/crates/dbt-adapter/src/record_batch.rs#L281-L317","documentation":"After arrow_json's Encoder writes a row into a byte buffer, the code converts those bytes to &str asserting the encoder only emits valid UTF-8. arrow_json's contract is to produce JSON text (UTF-8 by definition); a failure means the buffer contains non-UTF-8 bytes, which the library treats as an encoder-contract violation and panics via .expect.","triggerScenarios":"Calling encode_array_to_strings (via jsonify_nested_columns or jsonify_map_keys) where encoder.encode() writes bytes that std::str::from_utf8 rejects — e.g. a custom/patched encoder emitting raw non-UTF-8 payloads or a buffer mixing content from a non-UTF-8 source.","commonSituations":"Forked or outdated arrow-json versions whose encoder behavior deviates (e.g. emitting escaped latin-1 bytes); injecting a custom EncoderOptions/encoder implementation; memory corruption is essentially ruled out in safe Rust.","solutions":["Verify the arrow-json version matches the workspace pin and restore the stock encoder if it was patched.","Inspect the failing row's data: non-UTF-8 strings in nested columns should be sanitized by the driver before reaching the encoder.","Replace .expect with a lossy fallback (String::from_utf8_lossy) plus a warning if you must keep processing.","Report the offending data/row to maintainers; upstream arrow_json guarantees UTF-8 output."],"exampleFix":"// before\nlet s = std::str::from_utf8(&buf).expect(\"arrow_json::Encoder emits UTF-8\");\n// after\nlet s = std::str::from_utf8(&buf).map_err(|e| {\n    AdapterError::new(AdapterErrorKind::Internal, format!(\n        \"encoder emitted non-UTF-8 bytes: {e}\"))\n})?;","handlingStrategy":"fallback","validationCode":"// Pre-check is not possible on the internal buffer; validate output instead:\nfn is_utf8(buf: &[u8]) -> bool { std::str::from_utf8(buf).is_ok() }","typeGuard":null,"tryCatchPattern":"match std::str::from_utf8(&buf) {\n    Ok(s) => builder.append_value(s),\n    Err(_) => builder.append_value(String::from_utf8_lossy(&buf)), // or log & fail loudly\n}","preventionTips":["Use the stock arrow_json encoder — do not patch it to emit raw bytes","Sanitize non-UTF-8 string data at the driver boundary before it reaches JSON encoding","Add a round-trip test encoding representative nested rows and asserting UTF-8 validity","Pin arrow-json to the workspace-pinned version"],"tags":["arrow","arrow-json","utf-8","panic"],"backgroundTag":"internal-invariant-violation","analyzedSha":"0267ce9170576975b76b64ce856b2e5848e96617","analyzedAt":"2026-09-07T21:53:39.732Z","contentChangedAt":"2026-09-07T21:53:39.732Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}