AlexsJones/llmfit · error

embedded use_case_benchmarks.json is invalid

Error message

embedded use_case_benchmarks.json is invalid

What it means

task_bench.rs embeds data JSON via include_str! and lazily parses it into BenchFile { families: Vec<FamilyEntry> } inside a OnceLock. If the shipped use_case_benchmarks.json is malformed or its schema drifts (families array missing, score maps with non-numeric values), the .expect panics with this message on the first score() call. Like error 12, it is a data-integrity invariant that should be caught by tests before release.

Source

Thrown at llmfit-core/src/task_bench.rs:31

const TASK_BENCH_JSON: &str = include_str!("../data/use_case_benchmarks.json");

#[derive(serde::Deserialize)]
struct FamilyEntry {
    #[serde(rename = "match")]
    patterns: Vec<String>,
    scores: HashMap<String, f64>,
}

#[derive(serde::Deserialize)]
struct BenchFile {
    families: Vec<FamilyEntry>,
}

fn table() -> &'static [FamilyEntry] {
    static TABLE: OnceLock<Vec<FamilyEntry>> = OnceLock::new();
    TABLE.get_or_init(|| {
        serde_json::from_str::<BenchFile>(TASK_BENCH_JSON)
            .expect("embedded use_case_benchmarks.json is invalid")
            .families
    })
}

/// Benchmark score for a model on a task (`"coding"`, `"reasoning"`,
/// `"chat"`), or `None` if no family entry matches.
///
/// `name_lower` must already be lowercased. When several patterns match
/// (e.g. `qwen3` and `qwen3-coder`), the longest — most specific — wins.
pub fn score(name_lower: &str, task: &str) -> Option<f64> {
    let mut best: Option<(usize, f64)> = None;
    for entry in table() {
        for pattern in &entry.patterns {
            if name_lower.contains(pattern.as_str())
                && let Some(s) = entry.scores.get(task)
                && best.is_none_or(|(len, _)| pattern.len() > len)
            {
                best = Some((pattern.len(), *s));

View on GitHub (pinned to acc7e40c3a)

Solutions

1. Validate JSON syntax first: `python3 -m json.tool llmfit-core/data/use_case_benchmarks.json > /dev/null`
2. Align the file with the schema in task_bench.rs: a top-level { "families": [ { ... patterns, scores: {task: number} } ] } structure with numeric score values
3. Run `cargo test -p llmfit-core` (task bench tests exercise score()) after touching the data file or FamilyEntry

Example fix

# before (use_case_benchmarks.json) — trailing comma, string score
"families": [ { "patterns": ["qwen3"], "scores": { "coding": "0.82", } ] }
# => panic: embedded use_case_benchmarks.json is invalid

# after
"families": [ { "patterns": ["qwen3"], "scores": { "coding": 0.82 } } ]

import json
data = json.load(open("llmfit-core/data/use_case_benchmarks.json"))
assert isinstance(data.get("families"), list) and data["families"], "top-level 'families' list required"
for fam in data["families"]:
    assert isinstance(fam.get("patterns"), list), "patterns list required"
    assert all(isinstance(v, (int, float)) for v in fam["scores"].values()), "scores must be numeric"

When it happens

Trigger: Editing llmfit-core/data/use_case_benchmarks.json and leaving invalid JSON (trailing comma, unquoted key) or restructuring entries so serde cannot deserialize FamilyEntry (missing patterns/scores fields, string where f64 expected); the panic fires when any analysis path calls task_bench::score().

Common situations: Adding new model-family benchmark entries by hand; merging upstream changes to the data file; renaming struct fields in FamilyEntry without updating the JSON.

Understand the failure class

Background: JSON parse error: "Unexpected token" / "not valid JSON" / "failed to parse" — what JSON parsers are really complaining about — this error's family across 45 libraries.

Related errors

• embedded benchmarks.yaml is invalid
• JSON serialization failed
• valid MLX suffix regex
• CARGO_MANIFEST_DIR
• OUT_DIR

AI-assisted analysis of AlexsJones/llmfit@acc7e40c3a (2026-08-17). Data as JSON: /api/errors/30d2d95a651fdf20. Report an issue: GitHub.