{"record":{"id":"f1c1fc68319754e2","repo":"JuliusBrussee/caveman","slug":"invalid-fixture-path-q","errorCode":null,"errorMessage":"invalid fixture path %q","messagePattern":"invalid fixture path %q","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"engine/evals/harness.go","lineNumber":132,"sourceCode":"func RunDir(root string) (Report, error) {\n\tresolvedRoot, err := filepath.EvalSymlinks(root)\n\tif err != nil {\n\t\treturn Report{}, fmt.Errorf(\"resolve fixture directory: %w\", err)\n\t}\n\tmanifestRaw, err := os.ReadFile(filepath.Join(resolvedRoot, \"manifest.yaml\"))\n\tif err != nil {\n\t\treturn Report{}, fmt.Errorf(\"read manifest: %w\", err)\n\t}\n\tvar m Manifest\n\tif err := yaml.Unmarshal(manifestRaw, &m); err != nil {\n\t\treturn Report{}, fmt.Errorf(\"parse manifest: %w\", err)\n\t}\n\tif len(m.Fixtures) == 0 {\n\t\treturn Report{}, fmt.Errorf(\"manifest has no fixtures\")\n\t}\n\tread := func(name string) ([]byte, error) {\n\t\tif !fs.ValidPath(name) || name == \".\" {\n\t\t\treturn nil, fmt.Errorf(\"invalid fixture path %q\", name)\n\t\t}\n\t\tcandidate, err := filepath.EvalSymlinks(filepath.Join(resolvedRoot, filepath.FromSlash(name)))\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t\trel, err := filepath.Rel(resolvedRoot, candidate)\n\t\tif err != nil || rel == \"..\" || strings.HasPrefix(rel, \"..\"+string(filepath.Separator)) {\n\t\t\treturn nil, fmt.Errorf(\"fixture path escapes root: %q\", name)\n\t\t}\n\t\treturn os.ReadFile(candidate)\n\t}\n\treturn RunManifest(m, read)\n}\n\n// RunWithQuality replays the embedded fixture set and also sends baseline and\n// compressed prompts through a model runner before grading the answers.\nfunc RunWithQuality(ctx context.Context, opts QualityOptions) (Report, error) {\n\tm, err := EmbeddedManifest()","sourceCodeStart":114,"sourceCodeEnd":150,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/engine/evals/harness.go#L114-L150","documentation":"A manifest fixture path failed fs.ValidPath or was the bare '.'. The harness requires slash-separated, relative, unrooted paths (fs.FS semantics) before it will even attempt resolution, so absolute paths, backslash separators, empty segments, or '.' are rejected up front.","triggerScenarios":"A manifest entry with a path like '/fixtures/x.txt', 'C:\\fixtures\\x.txt', './x.txt', or '.' — anything that is not a clean relative slash path.","commonSituations":"Authoring manifests on Windows with native separators; copy-pasting absolute paths from an editor; tooling that emits './'-prefixed or empty paths.","solutions":["Rewrite fixture paths as plain relative slash paths: 'cases/smoke.txt' not '/cases/smoke.txt' or '.\\cases\\smoke.txt'.","Normalize paths in your generator: filepath.ToSlash + strip leading './' and '/'.","Lint the manifest for absolute or backslash paths before running evals.","Remember paths are resolved against the manifest's directory, so no root prefix is needed."],"exampleFix":"# before\n- name: smoke\n  path: /fixtures/smoke.txt\n\n# after\n- name: smoke\n  path: smoke.txt","handlingStrategy":"validation","validationCode":"func validFixturePath(name string) bool {\n    return fs.ValidPath(name) && name != \".\" && !strings.Contains(name, \"\\\\\")\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Author fixture paths as plain relative slash paths.","Normalize generated paths with filepath.ToSlash and strip leading './' or '/'.","Lint manifests for absolute paths and backslashes on Windows authoring."],"tags":["evals","path","manifest","validation"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}