{"record":{"id":"eb546f9ca2589d16","repo":"AlexsJones/llmfit","slug":"embedded-benchmarks-yaml-is-invalid","errorCode":null,"errorMessage":"embedded benchmarks.yaml is invalid","messagePattern":"embedded benchmarks\\.yaml is invalid","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"llmfit-core/src/quality.rs","lineNumber":548,"sourceCode":"            });\n        }\n    }\n\n    runner_ups.sort_by(|a, b| a.role.cmp(&b.role));\n    runner_ups\n}\n\n// ── YAML config loading ────────────────────────────────────────────\n\n/// Parse a YAML string into a `QualityConfig`.\npub fn load_quality_config(yaml: &str) -> Result<QualityConfig, String> {\n    yaml_serde::from_str(yaml).map_err(|e| format!(\"Failed to parse quality config: {}\", e))\n}\n\n/// Return the built-in default quality config (embedded from `data/benchmarks.yaml`).\npub fn default_quality_config() -> QualityConfig {\n    let yaml = include_str!(\"../data/benchmarks.yaml\");\n    load_quality_config(yaml).expect(\"embedded benchmarks.yaml is invalid\")\n}\n\n// ── Display helpers ────────────────────────────────────────────────\n\nimpl ModelQualityResult {\n    /// Print a human-readable summary of quality results.\n    pub fn display(&self) {\n        println!();\n        println!(\"  === Quality Benchmark Results ===\");\n        println!(\"  Model:    {}\", self.model);\n        println!(\"  Provider: {}\", self.provider);\n        println!();\n        println!(\n            \"  Overall:  quality={:.1}  speed={:.1} tok/s  composite={:.1}\",\n            self.overall_quality, self.overall_speed, self.overall_composite\n        );\n        println!();\n        println!(\"  Role             Quality  Speed    Composite  Tests\");","sourceCodeStart":530,"sourceCodeEnd":566,"githubUrl":"https://github.com/AlexsJones/llmfit/blob/acc7e40c3a0afbd36510a92f2f8f3d5177cfc0fe/llmfit-core/src/quality.rs#L530-L566","documentation":"default_quality_config() in llmfit-core/src/quality.rs embeds data/benchmarks.yaml via include_str! and parses it with serde_yml on first use, expecting a QualityConfig ({ roles: map of name -> { description, tests: [...] } }). If the shipped YAML is syntactically invalid or its shape no longer matches the structs, the .expect panics with this message. It fires lazily at the first quality command invocation, not at startup.","triggerScenarios":"Editing llmfit-core/data/benchmarks.yaml and introducing a YAML syntax error (bad indentation, tab characters, unclosed quote) or renaming/dropping the required roles/description/tests fields so deserialization fails; the panic then hits on the next `llmfit quality ...` run.","commonSituations":"Contributors adding new quality test definitions; refactoring QualityConfig/RoleDef/QualityTestDef structs (e.g. making a field required) without regenerating or updating the YAML; merge conflicts resolved badly in the data file.","solutions":["Validate the file before building: `python3 -c \"import yaml,sys; yaml.safe_load(open('llmfit-core/data/benchmarks.yaml'))\"` to catch syntax errors","Check the struct contract in quality.rs (QualityConfig.roles -> RoleDef { description, tests }) and make the YAML match it field-for-field","Run `cargo test -p llmfit-core quality` after any change to the YAML or the config structs so the failure surfaces in CI instead of at runtime"],"exampleFix":"# before (data/benchmarks.yaml) — 'description' missing, YAML misindented\nroles:\n  coder:\n      tests:\n        - name: inline_fn\n# => panic: embedded benchmarks.yaml is invalid\n\n# after\nroles:\n  coder:\n    description: Writes and reviews code\n    tests:\n      - name: inline_fn","handlingStrategy":"validation","validationCode":"import yaml\ncfg = yaml.safe_load(open(\"llmfit-core/data/benchmarks.yaml\"))\nassert set(cfg) == {\"roles\"} and cfg[\"roles\"], \"top-level 'roles' map required\"\nfor name, role in cfg[\"roles\"].items():\n    assert isinstance(role.get(\"description\"), str), f\"{name}: description string required\"\n    assert isinstance(role.get(\"tests\"), list) and role[\"tests\"], f\"{name}: tests list required\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate benchmarks.yaml with a schema check in CI (or cargo test that calls default_quality_config())","Regenerate/verify the YAML after any change to QualityConfig, RoleDef, or QualityTestDef","Resolve data-file merge conflicts by re-running the quality tests, not by trusting the merge"],"tags":["rust","yaml","embedded-data","serde","panic"],"backgroundTag":"yaml-parse-error","analyzedSha":"acc7e40c3a0afbd36510a92f2f8f3d5177cfc0fe","analyzedAt":"2026-08-17T10:35:29.658Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}