RyanCodrai/turbovec · error

invalid input value at vector

Error message

invalid input value at vector {vi}, coord {ci}: {v} (must be finite and |value| < 1e16 to avoid f32 norm overflow)

What it means

add() validates every coordinate before ingesting: values must be finite (no NaN/inf) and have magnitude < 1e16, because f32 norm computation would overflow otherwise. The first offending (vector index, coord index, value) is reported and add panics rather than silently corrupting the quantized index.

Solutions

1. Sanitize vectors before add: replace NaN/inf and clip values to |v| < 1e16.
2. Check the embedding pipeline for inputs that produce NaN (empty strings, tokenization failures).
3. Use a non-panicking add path if available (try_* variant) and handle the error per batch.

Example fix

// before
index.add(&raw_vectors); // panics: "invalid input value at vector 3, coord 7: NaN ..."
// after
let clean: Vec<f32> = raw_vectors.iter().map(|&v| if v.is_finite() && v.abs() < 1e16 { v } else { 0.0 }).collect();
index.add(&clean);

fn sanitize(v: &[f32]) -> Vec<f32> {
    v.iter().map(|&x| if x.is_finite() && x.abs() < 1e16 { x } else { 0.0 }).collect()
}

fn all_valid(v: &[f32]) -> bool {
    v.iter().all(|x| x.is_finite() && x.abs() < 1e16)
}

// panics cannot be caught idiomatically; avoid by validating
if !all_valid(&vectors) { return Err(AddError::InvalidValue); }
index.add(&vectors);

When it happens

Trigger: Calling add() on an index with a vectors buffer containing a NaN, +inf, -inf, or any |value| >= 1e16 at dim-aligned position (vi, ci); empty adds are no-ops and never trigger this.

Common situations: Upstream embedding model emitting NaN for degenerate/empty input text; uninitialized memory or zeroed-then-corrupted buffers; mixing f64 intermediate values with huge magnitudes before casting to f32.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• duplicate ids
• {e}
• {e}
• bit_width must be 2, 3, or 4, got
• duplicate id in batch

AI-assisted analysis of RyanCodrai/turbovec@ccab9f325e (2026-09-06). Data as JSON: /api/errors/c78bf8890863a40b. Report an issue: GitHub.

Appendix: source

Thrown at turbovec/src/lib.rs:733

    /// an error: it is stored with scale 0 and scores 0 against every
    /// query. See that constant for the rationale.
    pub fn add(&mut self, vectors: &[f32]) {
        let dim = self.dim.expect(
            "TurboQuantIndex dim is not set; use add_2d(vectors, dim) on the \
             first add or construct via TurboQuantIndex::new(dim, bit_width)",
        );
        let n = vectors.len() / dim;
        assert_eq!(
            vectors.len(),
            n * dim,
            "vectors length must be a multiple of dim"
        );
        // Empty add is a true no-op.
        if n == 0 {
            return;
        }
        if let Some((vi, ci, v)) = first_invalid_coord(vectors, dim) {
            panic!(
                "invalid input value at vector {vi}, coord {ci}: {v} \
                 (must be finite and |value| < 1e16 to avoid f32 norm overflow)",
            );
        }
        // One path, always. `add` reads the committed calibration and
        // never writes one — there is no warm-up buffer, no sample
        // threshold, and no batch that means more to the encoding than
        // any other. Whatever this index is calibrated to was set by an
        // explicit `calibrate` call, so a row's encoded bytes depend on
        // the row and the calibration and on nothing else: same rows,
        // same calibration, same bytes, however they were batched and in
        // whatever order they arrived.
        self.encode_and_append(vectors, n, dim);
    }

    /// Test-only switch that makes the next `encode` call panic, so tests
    /// can exercise the unwind guard below — and the ordering that guard
    /// depends on (#353). Panics inside `encode` are otherwise only

View on GitHub (pinned to ccab9f325e)