JuliusBrussee/caveman · error

quality runner is required

Error message

quality runner is required

What it means

Error "quality runner is required" thrown in JuliusBrussee/caveman.

Source

Thrown at engine/evals/harness.go:190

// ReadEmbeddedFixture reads one built-in fixture file.
func ReadEmbeddedFixture(name string) ([]byte, error) {
	return fixturesFS.ReadFile("fixtures/" + name)
}

// RunManifest replays an arbitrary manifest, reading fixture files via read.
// It is exported so tests can seed a deliberately-failing manifest and assert
// the gate fails closed.
func RunManifest(m Manifest, read func(name string) ([]byte, error)) (Report, error) {
	return runManifest(context.Background(), m, read, nil)
}

// RunManifestWithQuality is RunManifest plus the LLM-in-the-loop quality layer.
// The runner is injected so CI can use a deterministic local runner while release
// jobs can call a real gateway/model.
func RunManifestWithQuality(ctx context.Context, m Manifest, read func(name string) ([]byte, error), opts QualityOptions) (Report, error) {
	if opts.Runner == nil {
		return Report{}, fmt.Errorf("quality runner is required")
	}
	return runManifest(ctx, m, read, &opts)
}

// RunManifestWithSystem runs a caller-supplied system-under-test through the
// same graders and optional quality layer as the built-in engine.
func RunManifestWithSystem(ctx context.Context, m Manifest, read func(name string) ([]byte, error), system TransformRunner, opts *QualityOptions) (Report, error) {
	if system == nil {
		return Report{}, fmt.Errorf("system runner is required")
	}
	if opts != nil && opts.Runner == nil {
		return Report{}, fmt.Errorf("quality runner is required")
	}
	return runManifestWithSystem(ctx, m, read, system, opts)
}

func runManifest(ctx context.Context, m Manifest, read func(name string) ([]byte, error), quality *QualityOptions) (Report, error) {
	store, err := ccr.OpenMemory()

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Provide a quality runner to the eval harness.

When it happens

Trigger: Thrown at engine/evals/harness.go:190 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/dbd6b235205d16ea. Report an issue: GitHub.