cachebench: provider %q population %d cannot meet minimum el

Error message

cachebench: provider %q population %d cannot meet minimum eligible requests %d

What it means

During target validation, at least one provider's record count is below target.MinEligibleRequest. The benchmark requires each provider to have enough requests for a statistically meaningful cache-eligibility sample, so it refuses to run an underpopulated replay.

Source

Thrown at cacheengine/cachebench/replay.go:457

// sample gate even if every captured request proves cache eligible.
func ValidateReplayTarget(records []TraceRecord, target Target) error {
	if err := validateTarget(target); err != nil {
		return err
	}
	if len(records) == 0 {
		return errors.New("cachebench: replay trace is empty")
	}
	counts := map[string]int{}
	for _, record := range records {
		provider := strings.ToLower(strings.TrimSpace(record.Provider))
		if provider == "" {
			return fmt.Errorf("cachebench: replay request %q has empty provider", record.RequestID)
		}
		counts[provider]++
	}
	for provider, count := range counts {
		if count < target.MinEligibleRequest {
			return fmt.Errorf("cachebench: provider %q population %d cannot meet minimum eligible requests %d", provider, count, target.MinEligibleRequest)
		}
	}
	return nil
}

// Run validates full population, prepares equivalent bodies, then dispatches replay.
func (runner ReplayRunner) Run(ctx context.Context, records []TraceRecord, emit func(ReplayResult) error) error {
	if runner.Engine == nil || runner.Transport == nil || runner.Verifier == nil || emit == nil {
		return errors.New("cachebench: replay engine, transport, verifier, and emitter required")
	}
	if _, err := ValidateReplay(records, runner.Limits, runner.TimeScale); err != nil {
		return err
	}
	prepared, err := runner.prepare(ctx, records)
	if err != nil {
		return err
	}
	if runner.Target != nil {

View on GitHub (pinned to 27d5a3981a)

Solutions

Capture or include more trace records for the provider named in the message until count >= MinEligibleRequest.
Lower Target.MinEligibleRequest to a value every provider population can meet (fine for smoke tests).
Remove providers you do not intend to benchmark from the trace so their small populations no longer count.
Compute per-provider counts up front (group records by lowercased trimmed Provider) and reconcile against the target before calling Run.

Example fix

// before
target := Target{MinEligibleRequest: 10} // trace has 4 anthropic records

// after
target := Target{MinEligibleRequest: 4} // or add records until anthropic >= 10

Defensive patterns

Strategy: validation

Validate before calling

counts := map[string]int{}
for _, r := range records {
    counts[strings.ToLower(strings.TrimSpace(r.Provider))]++
}
for p, n := range counts {
    if n < target.MinEligibleRequest {
        return fmt.Errorf("provider %s underpopulated: %d < %d", p, n, target.MinEligibleRequest)
    }
}

Prevention

Compute per-provider counts and reconcile with MinEligibleRequest before calling Run.
Keep smoke-test traces paired with a low-minimum Target in test fixtures.
When splitting traces per provider, remember the minimum applies to each provider independently.

When it happens

Trigger: ValidateReplayTarget with, e.g., 3 records for "anthropic" while Target.MinEligibleRequest is 5. Triggered by small trace files, traces split per provider, or raising MinEligibleRequest above an existing population.

Common situations: Testing the harness with a tiny smoke trace; filtering a trace down to one model/provider and forgetting the minimum applies per provider; bumping MinEligibleRequest for a production run and replaying an old small trace.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/8276c5d878a387f2. Report an issue: GitHub.