What causes "Bytes support in dynamic fields is not yet implemented" in quickwit-oss/tantivy?

Declare a bytes field in the schema and store the value there

Bytes support in dynamic fields is not yet implemented

Error message

Bytes support in dynamic fields is not yet implemented

What it means

record_json_value_to_columnar_writer panics with unimplemented!() for Bytes leaves in dynamic JSON fields. The bytes recording call is commented out (TODO) pending support in the JSON utils path, so byte values cannot be turned into fast-field columns yet. Use schema-typed bytes fields instead.

Source

Thrown at src/fastfield/writer.rs:343

                columnar_writer.record_numerical(
                    doc,
                    json_path_writer.as_str(),
                    NumericalValue::from(val),
                );
            }
            ReferenceValueLeaf::Bool(val) => {
                columnar_writer.record_bool(doc, json_path_writer.as_str(), val);
            }
            ReferenceValueLeaf::Date(val) => {
                columnar_writer.record_datetime(doc, json_path_writer.as_str(), val);
            }
            ReferenceValueLeaf::Facet(_) => {
                unimplemented!("Facet support in dynamic fields is not yet implemented")
            }
            ReferenceValueLeaf::Bytes(_) => {
                // TODO: This can be re added once it is added to the JSON Utils section as well.
                // columnar_writer.record_bytes(doc, json_path_writer.as_str(), val);
                unimplemented!("Bytes support in dynamic fields is not yet implemented")
            }
            ReferenceValueLeaf::IpAddr(_) => {
                unimplemented!("IP address support in dynamic fields is not yet implemented")
            }
            ReferenceValueLeaf::PreTokStr(_) => {
                unimplemented!(
                    "Pre-tokenized string support in dynamic fields is not yet implemented"
                )
            }
            ReferenceValueLeaf::Custom(_) => {
                unimplemented!("the JSON field does not support custom field types")
            }
        },
        ReferenceValue::Array(elements) => {
            for el in elements {
                record_json_value_to_columnar_writer(
                    doc,
                    el,

View on GitHub (pinned to b5d8deb80c)

Solutions

Declare a bytes field in the schema and store the value there
Encode bytes as base64 strings in the JSON field if only retrieval (not fast-field access) is needed
Remove byte values from dynamic JSON documents before indexing

Example fix

// before
json_obj.insert("data", Value::Bytes(bytes));
// after
let data = schema_builder.add_bytes_field("data", STORED); // typed field

Defensive patterns

Strategy: validation

Validate before calling

fn json_has_bytes_leaves(v: &tantivy::schema::OwnedValue) -> bool {
    matches!(v, tantivy::schema::OwnedValue::Bytes(_))
}
// screen dynamic JSON values before add_document

Type guard

fn is_bytes_leaf(v: &tantivy::schema::Value) -> bool {
    matches!(v.as_value(), tantivy::schema::ReferenceValueLeaf::Bytes(_))
}

Try / catch

let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| writer.commit()));
if result.is_err() { eprintln!("segment write failed: bytes inside dynamic JSON fast fields"); }

Prevention

Use schema-typed bytes fields for binary values
Base64-encode binary data kept inside JSON fields
Validate documents before add_document/commit
Watch tantivy changelogs for bytes-in-JSON fast-field support

When it happens

Trigger: Writing a segment where a dynamic JSON field contains a Bytes leaf — via record_json_obj_to_columnar_writer or the recursive call during segment writing/test_columnar_from_jsons_aux.

Common situations: Storing binary data inside JSON fields; converting typed bytes fields to dynamic JSON fields.

Related errors

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/c5891d0fefb32087. Report an issue: GitHub.