quickwit-oss/tantivy · error · io::Error

InvalidData

InvalidData

Error message

No available applicable codec.

What it means

serialize_u64_based_column_values evaluates every registered codec's byte-size estimator against the column stats and picks the cheapest. If no codec's estimator returns Some (all returned None because none is applicable to the column's stats, e.g. cardinality/min-max ranges outside supported bounds), it raises InvalidData with 'No available applicable codec.' instead of writing a column with no codec.

Source

Thrown at columnar/src/column_values/u64_based/mod.rs:177

        let val_u64 = val.to_u64();
        stats_collector.collect(val_u64);
        for (_, estimator) in &mut estimators {
            estimator.collect(val_u64);
        }
    }
    for (_, estimator) in &mut estimators {
        estimator.finalize();
    }
    let stats = stats_collector.stats();
    let (_, best_codec, best_codec_estimator) = estimators
        .into_iter()
        .flat_map(|(codec_type, estimator)| {
            let num_bytes = estimator.estimate(&stats)?;
            Some((num_bytes, codec_type, estimator))
        })
        .min_by_key(|(num_bytes, _, _)| *num_bytes)
        .ok_or_else(|| {
            io::Error::new(io::ErrorKind::InvalidData, "No available applicable codec.")
        })?;
    best_codec.to_code().serialize(wrt)?;
    best_codec_estimator.serialize(
        &stats,
        &mut vals.boxed_iter().map(MonotonicallyMappableToU64::to_u64),
        wrt,
    )?;
    Ok(())
}

/// Load u64-based column values.
///
/// This method first identifies the codec off the first byte.
pub fn load_u64_based_column_values<T: MonotonicallyMappableToU64>(
    mut bytes: OwnedBytes,
) -> io::Result<Arc<dyn ColumnValues<T>>> {
    let codec_type: CodecType = bytes
        .first()

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Check the column's stats (num_vals, min/max) before serializing and skip or pad empty/degenerate columns
2. Ensure the default codec set is used (or include a fallback codec applicable to any stats) in the estimator list
3. Verify values fit the expected u64-mappable range before passing them to MonotonicallyMappableToU64-based serialization
4. Update to a version where an always-applicable fallback codec exists

Example fix

// before: serializing possibly-empty column
serialize_u64_based_column_values(vals.boxed_iter(), &codecs, wrt)?;
// after: guard empty columns
if vals.iter().next().is_none() { /* write empty column or skip */ } else {
    serialize_u64_based_column_values(vals.boxed_iter(), &codecs, wrt)?;
}

fn can_serialize<T: MonotonicallyMappableToU64>(vals: &[T], codecs: &[(CodecType, CodecEstimator)]) -> bool {
    let stats = PermutationStats::compute(&mut vals.iter().copied());
    codecs.iter().any(|(_, est)| est.estimate(&stats).is_some())
}
// guard: if !can_serialize(&vals, &codecs) { skip/pad column }

match serialize_u64_based_column_values(vals.boxed_iter(), &codecs, &mut w) {
    Ok(()) => {},
    Err(e) if e.kind() == io::ErrorKind::InvalidData && e.to_string().contains("No available applicable codec") => skip_or_pad_column(),
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling serialize_u64_based_column_values / serialize_column_mappable_to_u64 / serialize_multivalued_index (directly or via a writer) on a column whose stats make every codec estimator decline: typically a column with zero values, empty value range, or stats none of the codecs accept. Also hit in tests test_fastfield_gcd / test_serialize_and_load_simple when estimator lists are empty.

Common situations: Indexing an all-empty or degenerate fast field; building a custom codec list where no estimator supports the value range (e.g. values exceeding u64/GCD-able bounds); a version change removing or gating a previously-default codec so nothing applies to legacy column shapes.

Related errors

• InvalidData
• InvalidData
• InvalidData
• Invalid doc store version {v}
• No multivalued index is allowed when stacking column index

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/1009e895d11f9fd5. Report an issue: GitHub.