{"record":{"id":"30d2d95a651fdf20","repo":"AlexsJones/llmfit","slug":"embedded-use-case-benchmarks-json-is-invalid","errorCode":null,"errorMessage":"embedded use_case_benchmarks.json is invalid","messagePattern":"embedded use_case_benchmarks\\.json is invalid","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"llmfit-core/src/task_bench.rs","lineNumber":31,"sourceCode":"const TASK_BENCH_JSON: &str = include_str!(\"../data/use_case_benchmarks.json\");\n\n#[derive(serde::Deserialize)]\nstruct FamilyEntry {\n    #[serde(rename = \"match\")]\n    patterns: Vec<String>,\n    scores: HashMap<String, f64>,\n}\n\n#[derive(serde::Deserialize)]\nstruct BenchFile {\n    families: Vec<FamilyEntry>,\n}\n\nfn table() -> &'static [FamilyEntry] {\n    static TABLE: OnceLock<Vec<FamilyEntry>> = OnceLock::new();\n    TABLE.get_or_init(|| {\n        serde_json::from_str::<BenchFile>(TASK_BENCH_JSON)\n            .expect(\"embedded use_case_benchmarks.json is invalid\")\n            .families\n    })\n}\n\n/// Benchmark score for a model on a task (`\"coding\"`, `\"reasoning\"`,\n/// `\"chat\"`), or `None` if no family entry matches.\n///\n/// `name_lower` must already be lowercased. When several patterns match\n/// (e.g. `qwen3` and `qwen3-coder`), the longest — most specific — wins.\npub fn score(name_lower: &str, task: &str) -> Option<f64> {\n    let mut best: Option<(usize, f64)> = None;\n    for entry in table() {\n        for pattern in &entry.patterns {\n            if name_lower.contains(pattern.as_str())\n                && let Some(s) = entry.scores.get(task)\n                && best.is_none_or(|(len, _)| pattern.len() > len)\n            {\n                best = Some((pattern.len(), *s));","sourceCodeStart":13,"sourceCodeEnd":49,"githubUrl":"https://github.com/AlexsJones/llmfit/blob/acc7e40c3a0afbd36510a92f2f8f3d5177cfc0fe/llmfit-core/src/task_bench.rs#L13-L49","documentation":"task_bench.rs embeds data JSON via include_str! and lazily parses it into BenchFile { families: Vec<FamilyEntry> } inside a OnceLock. If the shipped use_case_benchmarks.json is malformed or its schema drifts (families array missing, score maps with non-numeric values), the .expect panics with this message on the first score() call. Like error 12, it is a data-integrity invariant that should be caught by tests before release.","triggerScenarios":"Editing llmfit-core/data/use_case_benchmarks.json and leaving invalid JSON (trailing comma, unquoted key) or restructuring entries so serde cannot deserialize FamilyEntry (missing patterns/scores fields, string where f64 expected); the panic fires when any analysis path calls task_bench::score().","commonSituations":"Adding new model-family benchmark entries by hand; merging upstream changes to the data file; renaming struct fields in FamilyEntry without updating the JSON.","solutions":["Validate JSON syntax first: `python3 -m json.tool llmfit-core/data/use_case_benchmarks.json > /dev/null`","Align the file with the schema in task_bench.rs: a top-level { \"families\": [ { ... patterns, scores: {task: number} } ] } structure with numeric score values","Run `cargo test -p llmfit-core` (task bench tests exercise score()) after touching the data file or FamilyEntry"],"exampleFix":"# before (use_case_benchmarks.json) — trailing comma, string score\n\"families\": [ { \"patterns\": [\"qwen3\"], \"scores\": { \"coding\": \"0.82\", } ] }\n# => panic: embedded use_case_benchmarks.json is invalid\n\n# after\n\"families\": [ { \"patterns\": [\"qwen3\"], \"scores\": { \"coding\": 0.82 } } ]","handlingStrategy":"validation","validationCode":"import json\ndata = json.load(open(\"llmfit-core/data/use_case_benchmarks.json\"))\nassert isinstance(data.get(\"families\"), list) and data[\"families\"], \"top-level 'families' list required\"\nfor fam in data[\"families\"]:\n    assert isinstance(fam.get(\"patterns\"), list), \"patterns list required\"\n    assert all(isinstance(v, (int, float)) for v in fam[\"scores\"].values()), \"scores must be numeric\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Run `python3 -m json.tool` on the data file before committing","Keep a cargo test that calls task_bench::score() so embedded-JSON breakage fails CI","Update the JSON and FamilyEntry struct in the same change, never separately"],"tags":["rust","json","embedded-data","serde","panic"],"backgroundTag":"json-parse-error","analyzedSha":"acc7e40c3a0afbd36510a92f2f8f3d5177cfc0fe","analyzedAt":"2026-08-17T10:35:29.658Z","schemaVersion":2},"datasetVersion":"2026-08-29T02:17:18.158Z"}