quickwit-oss/tantivy · critical

Invalid type code in JSON term

Error message

Invalid type code in JSON term

What it means

During JSON postings serialization, the writer reads term[0] as a type code and maps it via Type::from_code, panicking with "Invalid type code in JSON term" if the first byte is not a known Type code (src/postings/json_postings_writer.rs:84). JSON terms in tantivy are encoded with a type prefix byte (Str/Date/U64/I64/F64/Bool), so a corrupted or wrongly-prefixed term buffer breaks this invariant. This is an internal-consistency panic, normally unreachable from public APIs.

Source

Thrown at src/postings/json_postings_writer.rs:84

        doc_id_map: Option<&DocIdMapping>,
        ctx: &IndexingContext,
        serializer: &mut FieldSerializer,
    ) -> io::Result<()> {
        let mut term_buffer = JsonTermSerializer(Vec::with_capacity(48));
        let mut buffer_lender = BufferLender::default();
        let mut prev_term_id = u32::MAX;
        let mut term_path_len = 0; // this will be set in the first iteration
        for (_field, path_id, term, addr) in ordered_term_addrs {
            if prev_term_id != path_id.path_id() {
                term_buffer.clear();
                term_buffer.append_json_path(ordered_id_to_path[path_id.path_id() as usize]);
                term_path_len = term_buffer.len();
                prev_term_id = path_id.path_id();
            }
            term_buffer.truncate(term_path_len);
            term_buffer.append_bytes(term);

            let typ = Type::from_code(term[0]).expect("Invalid type code in JSON term");
            if typ == Type::Str {
                SpecializedPostingsWriter::<Rec>::serialize_one_term(
                    term_buffer.as_bytes(),
                    *addr,
                    doc_id_map,
                    &mut buffer_lender,
                    ctx,
                    serializer,
                )?;
            } else {
                SpecializedPostingsWriter::<DocIdRecorder>::serialize_one_term(
                    term_buffer.as_bytes(),
                    *addr,
                    doc_id_map,
                    &mut buffer_lender,
                    ctx,
                    serializer,
                )?;

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Delete and rebuild the index — corrupted segments are the most common cause and cannot be repaired in place.
2. Ensure all segments were written by the same tantivy version; do not mix data directories across versions.
3. If you build JSON terms manually, make the first byte a valid Type code matching the encoded value.
4. Reduce to a reproducible input document and file a bug with the tantivy maintainers if the index is freshly built.

// verify segment files were all written by the current tantivy version
// and pass integrity checks before search/merge

let ok = std::panic::catch_unwind(AssertUnwindSafe(|| merge_or_search()));
if ok.is_err() { rebuild_index_from_source()?; }

When it happens

Trigger: Indexing or merging JSON documents whose term dictionary produced a term whose first byte is not a valid type code; typically caused by a library bug, corrupted segment files during serialization/merge, or manually crafted terms inserted via low-level APIs.

Common situations: Segment corruption from truncated writes or bit-rot on disk; mixing segment files across incompatible tantivy versions; custom code building JSON-path terms with hand-set bytes; hitting a genuine serializer bug during merge of JSON fields.

Related errors

• Pre-tokenized string support in dynamic fields is not yet im
• Failed to acquire write lock on delete queue writer
• Empty DocOpstamp is forbidden
• This lock should never be poisoned
• Failed to acquire read lock on SegmentManager.

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/062fe15eaffd7323. Report an issue: GitHub.