JuliusBrussee/caveman · error

runner did not return input/output token usage

Error message

runner did not return input/output token usage

What it means

Error "runner did not return input/output token usage" thrown in JuliusBrussee/caveman.

Source

Thrown at engine/evals/quality.go:218

	if err != nil {
		return QualityReport{}, fmt.Errorf("compressed grading: %w", err)
	}
	report.BaselinePassed = len(report.BaselineFailures) == 0
	report.CompressedPassed = len(report.CompressedFailures) == 0
	if report.BaselinePassed && report.CompressedPassed {
		report.Retention = 1
	}
	return report, nil
}

func fillUsage(counter tokens.Counter, prompt []byte, resp ModelResponse, requireRunnerUsage bool) (ModelUsage, error) {
	usage := resp.Usage
	if usage.InputTokens > 0 && usage.OutputTokens > 0 {
		usage.Source = "runner"
		return usage, nil
	}
	if requireRunnerUsage {
		return ModelUsage{}, fmt.Errorf("runner did not return input/output token usage")
	}
	if usage.InputTokens == 0 {
		usage.InputTokens = counter.Count(prompt)
	}
	if usage.OutputTokens == 0 {
		usage.OutputTokens = counter.Count(resp.Output)
	}
	usage.Source = "local_tokenizer"
	return usage, nil
}

func gradeFailures(ctx context.Context, runner AnswerGrader, graders []Grader, subject Subject) ([]string, error) {
	var failures []string
	for _, g := range graders {
		v, err := runner.Grade(ctx, g, subject)
		if err != nil {
			return nil, err
		}

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Return input/output token usage from the runner.

When it happens

Trigger: Thrown at engine/evals/quality.go:218 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/a8c55cdff7e42cb3. Report an issue: GitHub.