JuliusBrussee/caveman · error

parse manifest: %w

Error message

parse manifest: %w

What it means

The embedded fixtures/manifest.yaml could not be parsed as YAML into the Manifest struct. This is a build-input problem: the shipped manifest no longer matches the Manifest schema (unknown/renamed fields with strict decoding, wrong types, or malformed YAML).

Source

Thrown at engine/evals/harness.go:103

	TokenRatio   float64         `json:"token_ratio"`
	WordsBefore  int             `json:"words_before"`
	WordsAfter   int             `json:"words_after"`
	WordRatio    float64         `json:"word_ratio"`
	Probes       ProbeSummary    `json:"probes"`
	Quality      *QualitySummary `json:"quality,omitempty"`
	Fixtures     []FixtureReport `json:"fixtures"`
}

// Run replays the embedded fixture set behind its graders and returns the
// report. It runs the engine's configured quality gate.
func Run() (Report, error) {
	var m Manifest
	raw, err := fixturesFS.ReadFile("fixtures/manifest.yaml")
	if err != nil {
		return Report{}, fmt.Errorf("read manifest: %w", err)
	}
	if err := yaml.Unmarshal(raw, &m); err != nil {
		return Report{}, fmt.Errorf("parse manifest: %w", err)
	}
	read := func(name string) ([]byte, error) {
		return fixturesFS.ReadFile("fixtures/" + name)
	}
	return RunManifest(m, read)
}

// RunDir replays a caller-supplied fixture directory containing manifest.yaml.
// Manifest file paths are slash-separated, relative, and confined to root even
// through symlinks; a custom empty corpus is a configuration error, never green.
func RunDir(root string) (Report, error) {
	resolvedRoot, err := filepath.EvalSymlinks(root)
	if err != nil {
		return Report{}, fmt.Errorf("resolve fixture directory: %w", err)
	}
	manifestRaw, err := os.ReadFile(filepath.Join(resolvedRoot, "manifest.yaml"))
	if err != nil {
		return Report{}, fmt.Errorf("read manifest: %w", err)

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Run `go run ./cmd/caveman-engine evals run` locally — the wrapped YAML error gives line/column.
2. Validate the YAML (yamllint or a parser) against the Manifest struct fields.
3. Regenerate or fix the manifest to match the current Manifest schema.
4. Add a CI step that runs the evals so manifest breakage is caught before release.

Example fix

# before: field renamed in schema
fixtures:
  - name: x
    file: x.txt

# after: match Manifest field tags exactly (check struct tags)
fixtures:
  - name: x
    path: x.txt

func manifestParses(path string) error {
    raw, err := os.ReadFile(path)
    if err != nil { return err }
    var m evals.Manifest
    return yaml.Unmarshal(raw, &m)
}

When it happens

Trigger: evals.Run() reading a manifest.yaml edited by hand or by a codegen step so that yaml.Unmarshal fails — bad indentation, a field typed differently than the struct (string where int expected), or duplicate keys.

Common situations: Editing fixtures/manifest.yaml without running the evals locally; a generator regenerating the manifest in a new format; schema evolution where fields were renamed without updating the manifest.

Related errors

• ${label}: row is missing a string "${field}"
• native pack: invalid identity
• read manifest: %w
• manifest has no fixtures
• invalid fixture path %q

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/569e31f96528ed34. Report an issue: GitHub.