JuliusBrussee/caveman · error

system runner is required

Error message

system runner is required

What it means

Error "system runner is required" thrown in JuliusBrussee/caveman.

Source

Thrown at engine/evals/harness.go:199

func RunManifest(m Manifest, read func(name string) ([]byte, error)) (Report, error) {
	return runManifest(context.Background(), m, read, nil)
}

// RunManifestWithQuality is RunManifest plus the LLM-in-the-loop quality layer.
// The runner is injected so CI can use a deterministic local runner while release
// jobs can call a real gateway/model.
func RunManifestWithQuality(ctx context.Context, m Manifest, read func(name string) ([]byte, error), opts QualityOptions) (Report, error) {
	if opts.Runner == nil {
		return Report{}, fmt.Errorf("quality runner is required")
	}
	return runManifest(ctx, m, read, &opts)
}

// RunManifestWithSystem runs a caller-supplied system-under-test through the
// same graders and optional quality layer as the built-in engine.
func RunManifestWithSystem(ctx context.Context, m Manifest, read func(name string) ([]byte, error), system TransformRunner, opts *QualityOptions) (Report, error) {
	if system == nil {
		return Report{}, fmt.Errorf("system runner is required")
	}
	if opts != nil && opts.Runner == nil {
		return Report{}, fmt.Errorf("quality runner is required")
	}
	return runManifestWithSystem(ctx, m, read, system, opts)
}

func runManifest(ctx context.Context, m Manifest, read func(name string) ([]byte, error), quality *QualityOptions) (Report, error) {
	store, err := ccr.OpenMemory()
	if err != nil {
		return Report{}, fmt.Errorf("open recovery store: %w", err)
	}
	defer store.Close()
	eng := engine.New(store, nil)
	return runManifestWithSystem(ctx, m, read, engineSystem{eng: eng}, quality)
}

func runManifestWithSystem(ctx context.Context, m Manifest, read func(name string) ([]byte, error), system TransformRunner, quality *QualityOptions) (Report, error) {

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Provide a system runner to the eval harness.

When it happens

Trigger: Thrown at engine/evals/harness.go:199 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/1772bc786c09603d. Report an issue: GitHub.