JuliusBrussee/caveman · error

quality fixture %q: %w

Error message

quality fixture %q: %w

What it means

Error "quality fixture %q: %w" thrown in JuliusBrussee/caveman.

Source

Thrown at engine/evals/harness.go:276

		mergeProbeSummary(&report.Probes, probeSummary)
		// A fixture with no graders is a configuration error, not a free pass.
		if len(f.Graders) == 0 {
			fr.Passed = false
			fr.Failures = append(fr.Failures, "no graders configured")
		}
		for _, g := range f.Graders {
			if v := Grade(g, subject); !v.Passed {
				fr.Passed = false
				fr.Failures = append(fr.Failures, v.Reason)
			}
		}
		if !fr.Passed {
			report.Passed = false
		}
		if quality != nil {
			qr, err := runQuality(ctx, *quality, f, input, res.Output, res.ContentType, res.Ratio, res.PassedThrough)
			if err != nil {
				return Report{}, fmt.Errorf("quality fixture %q: %w", f.Name, err)
			}
			fr.Quality = &qr
		}
		report.TokensBefore += fr.TokensBefore
		report.TokensAfter += fr.TokensAfter
		report.WordsBefore += fr.WordsBefore
		report.WordsAfter += fr.WordsAfter
		report.Fixtures = append(report.Fixtures, fr)
	}
	report.TokenRatio = ratio(report.TokensBefore, report.TokensAfter)
	report.WordRatio = ratio(report.WordsBefore, report.WordsAfter)
	if quality != nil {
		report.Quality = summarizeQuality(report.Fixtures, *quality)
		if !report.Quality.Passed {
			report.Passed = false
		}
	}
	return report, nil

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Fix the quality fixture run error and retry.

When it happens

Trigger: Thrown at engine/evals/harness.go:276 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/4ad03b3939790d23. Report an issue: GitHub.