AlexsJones/llmfit · error

embedded benchmarks.yaml is invalid

Error message

embedded benchmarks.yaml is invalid

What it means

default_quality_config() in llmfit-core/src/quality.rs embeds data/benchmarks.yaml via include_str! and parses it with serde_yml on first use, expecting a QualityConfig ({ roles: map of name -> { description, tests: [...] } }). If the shipped YAML is syntactically invalid or its shape no longer matches the structs, the .expect panics with this message. It fires lazily at the first quality command invocation, not at startup.

Source

Thrown at llmfit-core/src/quality.rs:548

            });
        }
    }

    runner_ups.sort_by(|a, b| a.role.cmp(&b.role));
    runner_ups
}

// ── YAML config loading ────────────────────────────────────────────

/// Parse a YAML string into a `QualityConfig`.
pub fn load_quality_config(yaml: &str) -> Result<QualityConfig, String> {
    yaml_serde::from_str(yaml).map_err(|e| format!("Failed to parse quality config: {}", e))
}

/// Return the built-in default quality config (embedded from `data/benchmarks.yaml`).
pub fn default_quality_config() -> QualityConfig {
    let yaml = include_str!("../data/benchmarks.yaml");
    load_quality_config(yaml).expect("embedded benchmarks.yaml is invalid")
}

// ── Display helpers ────────────────────────────────────────────────

impl ModelQualityResult {
    /// Print a human-readable summary of quality results.
    pub fn display(&self) {
        println!();
        println!("  === Quality Benchmark Results ===");
        println!("  Model:    {}", self.model);
        println!("  Provider: {}", self.provider);
        println!();
        println!(
            "  Overall:  quality={:.1}  speed={:.1} tok/s  composite={:.1}",
            self.overall_quality, self.overall_speed, self.overall_composite
        );
        println!();
        println!("  Role             Quality  Speed    Composite  Tests");

View on GitHub (pinned to acc7e40c3a)

Solutions

1. Validate the file before building: `python3 -c "import yaml,sys; yaml.safe_load(open('llmfit-core/data/benchmarks.yaml'))"` to catch syntax errors
2. Check the struct contract in quality.rs (QualityConfig.roles -> RoleDef { description, tests }) and make the YAML match it field-for-field
3. Run `cargo test -p llmfit-core quality` after any change to the YAML or the config structs so the failure surfaces in CI instead of at runtime

Example fix

# before (data/benchmarks.yaml) — 'description' missing, YAML misindented
roles:
  coder:
      tests:
        - name: inline_fn
# => panic: embedded benchmarks.yaml is invalid

# after
roles:
  coder:
    description: Writes and reviews code
    tests:
      - name: inline_fn

import yaml
cfg = yaml.safe_load(open("llmfit-core/data/benchmarks.yaml"))
assert set(cfg) == {"roles"} and cfg["roles"], "top-level 'roles' map required"
for name, role in cfg["roles"].items():
    assert isinstance(role.get("description"), str), f"{name}: description string required"
    assert isinstance(role.get("tests"), list) and role["tests"], f"{name}: tests list required"

When it happens

Trigger: Editing llmfit-core/data/benchmarks.yaml and introducing a YAML syntax error (bad indentation, tab characters, unclosed quote) or renaming/dropping the required roles/description/tests fields so deserialization fails; the panic then hits on the next `llmfit quality ...` run.

Common situations: Contributors adding new quality test definitions; refactoring QualityConfig/RoleDef/QualityTestDef structs (e.g. making a field required) without regenerating or updating the YAML; merge conflicts resolved badly in the data file.

Related errors

• embedded use_case_benchmarks.json is invalid
• JSON serialization failed
• valid MLX suffix regex
• CARGO_MANIFEST_DIR
• OUT_DIR

AI-assisted analysis of AlexsJones/llmfit@acc7e40c3a (2026-08-17). Data as JSON: /api/errors/eb546f9ca2589d16. Report an issue: GitHub.