JuliusBrussee/caveman · error

manifest has no fixtures

Error message

manifest has no fixtures

What it means

RunDir treats an empty fixture list as a configuration error by design: a manifest that parses but declares zero fixtures would trivially pass the gate, so the harness fails closed instead of reporting green on nothing. This prevents a silent no-op eval run from looking like a success.

Source

Thrown at engine/evals/harness.go:128

// RunDir replays a caller-supplied fixture directory containing manifest.yaml.
// Manifest file paths are slash-separated, relative, and confined to root even
// through symlinks; a custom empty corpus is a configuration error, never green.
func RunDir(root string) (Report, error) {
	resolvedRoot, err := filepath.EvalSymlinks(root)
	if err != nil {
		return Report{}, fmt.Errorf("resolve fixture directory: %w", err)
	}
	manifestRaw, err := os.ReadFile(filepath.Join(resolvedRoot, "manifest.yaml"))
	if err != nil {
		return Report{}, fmt.Errorf("read manifest: %w", err)
	}
	var m Manifest
	if err := yaml.Unmarshal(manifestRaw, &m); err != nil {
		return Report{}, fmt.Errorf("parse manifest: %w", err)
	}
	if len(m.Fixtures) == 0 {
		return Report{}, fmt.Errorf("manifest has no fixtures")
	}
	read := func(name string) ([]byte, error) {
		if !fs.ValidPath(name) || name == "." {
			return nil, fmt.Errorf("invalid fixture path %q", name)
		}
		candidate, err := filepath.EvalSymlinks(filepath.Join(resolvedRoot, filepath.FromSlash(name)))
		if err != nil {
			return nil, err
		}
		rel, err := filepath.Rel(resolvedRoot, candidate)
		if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
			return nil, fmt.Errorf("fixture path escapes root: %q", name)
		}
		return os.ReadFile(candidate)
	}
	return RunManifest(m, read)
}

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Ensure the manifest declares at least one fixture under the fixtures key.
2. If you intentionally want an empty corpus, this harness does not support it — use a different entry point or add a placeholder fixture.
3. Check that fixture entries use the correct field name so they actually deserialize (otherwise the list stays empty).
4. Print the parsed manifest in a dry-run to confirm fixtures are being read.

Example fix

# before
fixtures: []

# after
fixtures:
  - name: smoke
    path: smoke.txt

func manifestHasFixtures(path string) bool {
    raw, err := os.ReadFile(path)
    if err != nil { return false }
    var m evals.Manifest
    return yaml.Unmarshal(raw, &m) == nil && len(m.Fixtures) > 0
}

When it happens

Trigger: evals.RunDir (or evals run --fixtures DIR) with a manifest.yaml whose fixtures list is empty, absent, or commented out.

Common situations: A filtering step or template renders an empty fixtures array; commenting out all fixtures during debugging and forgetting to restore; a corpus where fixture entries were moved to another field name so none deserialize.

Related errors

• caveman build: requiredFixturePassRate must be in (0,1]
• read manifest: %w
• parse manifest: %w
• invalid fixture path %q
• cave_budget_denomination_ambiguous

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/6080b2689338e1ad. Report an issue: GitHub.