quickwit-oss/tantivy · error

support only up to u16::MAX field ids

Error message

support only up to u16::MAX field ids

What it means

This panic comes from `Document::add_field_value` in tantivy's default document implementation. Each (field, value) pair is stored in a compact `FieldValueAddr` struct where the field id is packed into a `u16` to keep the document representation memory-efficient. If the schema assigns the field an id greater than `u16::MAX` (65535), the `try_into::<u16>()` conversion fails and the `.expect` panics.

Source

Thrown at src/schema/document/default_document.rs:143

    pub fn add_custom(&mut self, field: Field, value: &[u8]) {
        self.add_leaf_field_value(field, ReferenceValueLeaf::Custom(value));
    }

    /// Add a dynamic object field
    pub fn add_object(&mut self, field: Field, object: BTreeMap<String, OwnedValue>) {
        self.add_field_value(field, &OwnedValue::from(object));
    }

    /// Add a (field, value) to the document.
    ///
    /// `OwnedValue` implements Value, which should be easiest to use, but is not the most
    /// performant.
    pub fn add_field_value<'a, V: Value<'a>>(&mut self, field: Field, value: V) {
        let field_value = FieldValueAddr {
            field: field
                .field_id()
                .try_into()
                .expect("support only up to u16::MAX field ids"),
            value_addr: self.add_value(value),
        };
        self.field_values.push(field_value);
    }

    /// Add a (field, leaf value) to the document.
    /// Leaf values don't have nested values.
    pub fn add_leaf_field_value<'a, T: Into<ReferenceValueLeaf<'a>>>(
        &mut self,
        field: Field,
        typed_val: T,
    ) {
        let value = typed_val.into();
        let field_value = FieldValueAddr {
            field: field
                .field_id()
                .try_into()
                .expect("support only up to u16::MAX field ids"),

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Check the schema size before indexing: ensure the number of fields added to `SchemaBuilder` stays at or below 65536 and fail early with a clear application-level error.
2. Remap high-cardinality keys to a smaller set of schema fields, storing the original key inside a JSON/text field instead of creating one schema field per key.
3. If you genuinely need >65535 fields, split the data across multiple indexes/schemas so each schema stays under the limit.
4. Wrap document construction in `catch_unwind` only as a last resort; this is a hard limit of the format, not a recoverable condition.

Example fix

// before: one field per JSON key, blows past u16::MAX
for key in json_keys { schema_builder.add_text_field(key, TEXT); }

// after: bounded schema + dynamic key stored in a single field
let dynamic = schema_builder.add_json_field("attributes", STORED);
let mut doc = Document::default();
doc.add_field_value(dynamic, serde_json::to_value(json_obj)?);

fn validate_schema(schema: &tantivy::schema::Schema) -> Result<(), String> {
    if schema.num_fields() > (u16::MAX as usize) + 1 {
        return Err(format!(
            "schema has {} fields; tantivy supports at most {} field ids (u16::MAX)",
            schema.num_fields(),
            u16::MAX
        ));
    }
    Ok(())
}
validate_schema(&schema)?;

fn field_id_fits_u16(field: &tantivy::schema::Field) -> bool {
    (field.field_id() as usize) <= u16::MAX as usize
}

// expect() panics, not Result: isolate doc construction
let doc = std::panic::catch_unwind(|| {
    let mut doc = Document::default();
    doc.add_field_value(field, value);
    doc
})
.map_err(|_| anyhow::anyhow!("field id exceeds u16::MAX limit"))?;

When it happens

Trigger: Calling `Document::add_field_value(field, value)` (or any higher-level `add_*(field, value)` convenience such as `add_text`, `add_u64`, `add_date`) with a `Field` whose id exceeds 65535, i.e. a schema with more than 65536 fields.

Common situations: Programmatically generating a schema with hundreds of thousands of fields (e.g. one field per key in user-supplied JSON), building a schema from unbounded/log-columnar data, or reusing/multiplexing indexes across tenants where field ids grow unchecked.

Related errors

• Field already exists in schema {}
• Failed to acquire write lock on delete queue writer
• Empty DocOpstamp is forbidden
• This lock should never be poisoned
• Failed to acquire read lock on SegmentManager.

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/dd1eb2561fc67eeb. Report an issue: GitHub.