JuliusBrussee/caveman · error

answer-key runner requires expected answer

Error message

answer-key runner requires expected answer

What it means

Error "answer-key runner requires expected answer" thrown in JuliusBrussee/caveman.

Source

Thrown at engine/evals/quality.go:339

	return []byte("Use the context to answer the question.\n\nContext:\n" + string(context) + "\n\nQuestion:\n" + strings.TrimSpace(question) + "\n\nAnswer only.")
}

func expectedAnswer(graders []Grader) string {
	for _, g := range graders {
		if g.Type == "exact_match" {
			return graderString(g)
		}
	}
	return ""
}

// AnswerKeyModel is an explicit offline smoke runner. It never calls a provider;
// it returns the fixture's expected answer and labels usage as local.
type AnswerKeyModel struct{}

func (AnswerKeyModel) Complete(_ context.Context, req ModelRequest) (ModelResponse, error) {
	if strings.TrimSpace(req.ExpectedAnswer) == "" {
		return ModelResponse{}, fmt.Errorf("answer-key runner requires expected answer")
	}
	return ModelResponse{Output: []byte(req.ExpectedAnswer)}, nil
}

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Provide the expected answer for the answer-key runner.

When it happens

Trigger: Thrown at engine/evals/quality.go:339 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/46102e446625be4e. Report an issue: GitHub.