JuliusBrussee/caveman · error

invalid fixture path %q

Error message

invalid fixture path %q

What it means

A manifest fixture path failed fs.ValidPath or was the bare '.'. The harness requires slash-separated, relative, unrooted paths (fs.FS semantics) before it will even attempt resolution, so absolute paths, backslash separators, empty segments, or '.' are rejected up front.

Source

Thrown at engine/evals/harness.go:132

func RunDir(root string) (Report, error) {
	resolvedRoot, err := filepath.EvalSymlinks(root)
	if err != nil {
		return Report{}, fmt.Errorf("resolve fixture directory: %w", err)
	}
	manifestRaw, err := os.ReadFile(filepath.Join(resolvedRoot, "manifest.yaml"))
	if err != nil {
		return Report{}, fmt.Errorf("read manifest: %w", err)
	}
	var m Manifest
	if err := yaml.Unmarshal(manifestRaw, &m); err != nil {
		return Report{}, fmt.Errorf("parse manifest: %w", err)
	}
	if len(m.Fixtures) == 0 {
		return Report{}, fmt.Errorf("manifest has no fixtures")
	}
	read := func(name string) ([]byte, error) {
		if !fs.ValidPath(name) || name == "." {
			return nil, fmt.Errorf("invalid fixture path %q", name)
		}
		candidate, err := filepath.EvalSymlinks(filepath.Join(resolvedRoot, filepath.FromSlash(name)))
		if err != nil {
			return nil, err
		}
		rel, err := filepath.Rel(resolvedRoot, candidate)
		if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
			return nil, fmt.Errorf("fixture path escapes root: %q", name)
		}
		return os.ReadFile(candidate)
	}
	return RunManifest(m, read)
}

// RunWithQuality replays the embedded fixture set and also sends baseline and
// compressed prompts through a model runner before grading the answers.
func RunWithQuality(ctx context.Context, opts QualityOptions) (Report, error) {
	m, err := EmbeddedManifest()

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Rewrite fixture paths as plain relative slash paths: 'cases/smoke.txt' not '/cases/smoke.txt' or '.\cases\smoke.txt'.
2. Normalize paths in your generator: filepath.ToSlash + strip leading './' and '/'.
3. Lint the manifest for absolute or backslash paths before running evals.
4. Remember paths are resolved against the manifest's directory, so no root prefix is needed.

Example fix

# before
- name: smoke
  path: /fixtures/smoke.txt

# after
- name: smoke
  path: smoke.txt

func validFixturePath(name string) bool {
    return fs.ValidPath(name) && name != "." && !strings.Contains(name, "\\")
}

When it happens

Trigger: A manifest entry with a path like '/fixtures/x.txt', 'C:\fixtures\x.txt', './x.txt', or '.' — anything that is not a clean relative slash path.

Common situations: Authoring manifests on Windows with native separators; copy-pasting absolute paths from an editor; tooling that emits './'-prefixed or empty paths.

Related errors

• caveman build: requiredFixturePassRate must be in (0,1]
• File not found: {filepath}
• native pack: invalid identity
• repository intelligence: cwd is not a directory
• read manifest: %w

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/f1c1fc68319754e2. Report an issue: GitHub.