JuliusBrussee/caveman · error

read manifest: %w

Error message

read manifest: %w

What it means

evals.Run() could not read the embedded fixture manifest at fixtures/manifest.yaml from the compiled-in filesystem. Because the fixtures are embedded at build time, this error almost always means the binary was built from a tree where the embed directory was missing or the embed directive drifted.

Source

Thrown at engine/evals/harness.go:100

	Passed       bool            `json:"passed"`
	TokensBefore int             `json:"tokens_before"`
	TokensAfter  int             `json:"tokens_after"`
	TokenRatio   float64         `json:"token_ratio"`
	WordsBefore  int             `json:"words_before"`
	WordsAfter   int             `json:"words_after"`
	WordRatio    float64         `json:"word_ratio"`
	Probes       ProbeSummary    `json:"probes"`
	Quality      *QualitySummary `json:"quality,omitempty"`
	Fixtures     []FixtureReport `json:"fixtures"`
}

// Run replays the embedded fixture set behind its graders and returns the
// report. It runs the engine's configured quality gate.
func Run() (Report, error) {
	var m Manifest
	raw, err := fixturesFS.ReadFile("fixtures/manifest.yaml")
	if err != nil {
		return Report{}, fmt.Errorf("read manifest: %w", err)
	}
	if err := yaml.Unmarshal(raw, &m); err != nil {
		return Report{}, fmt.Errorf("parse manifest: %w", err)
	}
	read := func(name string) ([]byte, error) {
		return fixturesFS.ReadFile("fixtures/" + name)
	}
	return RunManifest(m, read)
}

// RunDir replays a caller-supplied fixture directory containing manifest.yaml.
// Manifest file paths are slash-separated, relative, and confined to root even
// through symlinks; a custom empty corpus is a configuration error, never green.
func RunDir(root string) (Report, error) {
	resolvedRoot, err := filepath.EvalSymlinks(root)
	if err != nil {
		return Report{}, fmt.Errorf("resolve fixture directory: %w", err)
	}

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Verify engine/evals/fixtures/manifest.yaml exists in the source tree you built from.
2. Force a clean rebuild (go clean -cache; go build) to rule out stale embedded data.
3. If you forked the repo, restore the fixtures directory or point RunDir at an external fixture root instead.
4. Check the embed directive still matches the fixtures path.

// At build-check time, confirm the embedded fixtures exist
if _, err := fixturesFS.ReadFile("fixtures/manifest.yaml"); err != nil {
    log.Fatal("embedded fixtures missing — rebuild from a complete source tree")
}

When it happens

Trigger: Calling evals.Run() (or the evals run CLI subcommand) in a build where the fixtures embed FS has no fixtures/manifest.yaml — wrong build tags, moved fixture directory, or a stale/partial build cache.

Common situations: Building after moving engine/evals/fixtures; go build cache serving a stale embed; vendoring/forking that dropped the fixtures directory; empty-directory tricks used to satisfy embed failing for the manifest specifically.

Related errors

• caveman build: requiredFixturePassRate must be in (0,1]
• parse manifest: %w
• manifest has no fixtures
• invalid fixture path %q
• caveman build: dataResidency is not enforced yet; refusing t

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/065e22f50d30c4d4. Report an issue: GitHub.