JuliusBrussee/caveman · error

baseline grading: %w

Error message

baseline grading: %w

What it means

Error "baseline grading: %w" thrown in JuliusBrussee/caveman.

Source

Thrown at engine/evals/quality.go:197

	counter := tokens.Default()
	report.BaselineUsage, err = fillUsage(counter, baselinePrompt, baseline, opts.RequireUsage)
	if err != nil {
		return QualityReport{}, fmt.Errorf("baseline usage: %w", err)
	}
	report.CompressedUsage, err = fillUsage(counter, compressedPrompt, compressedResp, opts.RequireUsage)
	if err != nil {
		return QualityReport{}, fmt.Errorf("compressed usage: %w", err)
	}

	baseSubject := Subject{Input: input, Output: baseline.Output, ContentType: contentType, PassedThrough: true}
	compSubject := Subject{Input: input, Output: compressedResp.Output, Ratio: compressionRatio, ContentType: contentType, PassedThrough: passedThrough}
	grader := opts.Grader
	if grader == nil {
		grader = LocalGrader{}
	}
	report.BaselineFailures, err = gradeFailures(ctx, grader, graders, baseSubject)
	if err != nil {
		return QualityReport{}, fmt.Errorf("baseline grading: %w", err)
	}
	report.CompressedFailures, err = gradeFailures(ctx, grader, graders, compSubject)
	if err != nil {
		return QualityReport{}, fmt.Errorf("compressed grading: %w", err)
	}
	report.BaselinePassed = len(report.BaselineFailures) == 0
	report.CompressedPassed = len(report.CompressedFailures) == 0
	if report.BaselinePassed && report.CompressedPassed {
		report.Retention = 1
	}
	return report, nil
}

func fillUsage(counter tokens.Counter, prompt []byte, resp ModelResponse, requireRunnerUsage bool) (ModelUsage, error) {
	usage := resp.Usage
	if usage.InputTokens > 0 && usage.OutputTokens > 0 {
		usage.Source = "runner"
		return usage, nil

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Fix the baseline grading call and retry.

When it happens

Trigger: Thrown at engine/evals/quality.go:197 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/b4c63a10398fa2a5. Report an issue: GitHub.