{"record":{"id":"6080b2689338e1ad","repo":"JuliusBrussee/caveman","slug":"manifest-has-no-fixtures","errorCode":null,"errorMessage":"manifest has no fixtures","messagePattern":"manifest has no fixtures","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"engine/evals/harness.go","lineNumber":128,"sourceCode":"\n// RunDir replays a caller-supplied fixture directory containing manifest.yaml.\n// Manifest file paths are slash-separated, relative, and confined to root even\n// through symlinks; a custom empty corpus is a configuration error, never green.\nfunc RunDir(root string) (Report, error) {\n\tresolvedRoot, err := filepath.EvalSymlinks(root)\n\tif err != nil {\n\t\treturn Report{}, fmt.Errorf(\"resolve fixture directory: %w\", err)\n\t}\n\tmanifestRaw, err := os.ReadFile(filepath.Join(resolvedRoot, \"manifest.yaml\"))\n\tif err != nil {\n\t\treturn Report{}, fmt.Errorf(\"read manifest: %w\", err)\n\t}\n\tvar m Manifest\n\tif err := yaml.Unmarshal(manifestRaw, &m); err != nil {\n\t\treturn Report{}, fmt.Errorf(\"parse manifest: %w\", err)\n\t}\n\tif len(m.Fixtures) == 0 {\n\t\treturn Report{}, fmt.Errorf(\"manifest has no fixtures\")\n\t}\n\tread := func(name string) ([]byte, error) {\n\t\tif !fs.ValidPath(name) || name == \".\" {\n\t\t\treturn nil, fmt.Errorf(\"invalid fixture path %q\", name)\n\t\t}\n\t\tcandidate, err := filepath.EvalSymlinks(filepath.Join(resolvedRoot, filepath.FromSlash(name)))\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t\trel, err := filepath.Rel(resolvedRoot, candidate)\n\t\tif err != nil || rel == \"..\" || strings.HasPrefix(rel, \"..\"+string(filepath.Separator)) {\n\t\t\treturn nil, fmt.Errorf(\"fixture path escapes root: %q\", name)\n\t\t}\n\t\treturn os.ReadFile(candidate)\n\t}\n\treturn RunManifest(m, read)\n}\n","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/engine/evals/harness.go#L110-L146","documentation":"RunDir treats an empty fixture list as a configuration error by design: a manifest that parses but declares zero fixtures would trivially pass the gate, so the harness fails closed instead of reporting green on nothing. This prevents a silent no-op eval run from looking like a success.","triggerScenarios":"evals.RunDir (or evals run --fixtures DIR) with a manifest.yaml whose fixtures list is empty, absent, or commented out.","commonSituations":"A filtering step or template renders an empty fixtures array; commenting out all fixtures during debugging and forgetting to restore; a corpus where fixture entries were moved to another field name so none deserialize.","solutions":["Ensure the manifest declares at least one fixture under the fixtures key.","If you intentionally want an empty corpus, this harness does not support it — use a different entry point or add a placeholder fixture.","Check that fixture entries use the correct field name so they actually deserialize (otherwise the list stays empty).","Print the parsed manifest in a dry-run to confirm fixtures are being read."],"exampleFix":"# before\nfixtures: []\n\n# after\nfixtures:\n  - name: smoke\n    path: smoke.txt","handlingStrategy":"validation","validationCode":"func manifestHasFixtures(path string) bool {\n    raw, err := os.ReadFile(path)\n    if err != nil { return false }\n    var m evals.Manifest\n    return yaml.Unmarshal(raw, &m) == nil && len(m.Fixtures) > 0\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never commit a manifest with an empty fixtures list that you expect to run.","Confirm fixture entries land under the fixtures key with the right field names.","Add a dry-run print of parsed fixture count before grading."],"tags":["evals","manifest","config","fail-closed"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}