JuliusBrussee/caveman · error

cachebench: provider population exceeds 1024

Error message

cachebench: provider population exceeds 1024

What it means

EvaluateObserved groups records by lowercased trimmed Provider field and caps distinct providers at 1024, mirroring the corpus path's population bound. More than 1024 distinct provider strings indicates dirty data (unnormalized names, per-request IDs in the field) rather than a real provider set.

Source

Thrown at cacheengine/cachebench/observed.go:167

// EvaluateObserved evaluates supplied observations without completeness claim.
func EvaluateObserved(records []ObservationRecord, target Target) (Report, error) {
	if err := validateTarget(target); err != nil {
		return Report{}, err
	}
	if len(records) == 0 {
		return Report{}, errors.New("cachebench: no observation records")
	}
	if err := validateObservationRecords(records); err != nil {
		return Report{}, err
	}
	groups := map[string][]ObservationRecord{}
	for _, record := range records {
		provider := strings.ToLower(strings.TrimSpace(record.Provider))
		groups[provider] = append(groups[provider], record)
	}
	if len(groups) > 1024 {
		return Report{}, errors.New("cachebench: provider population exceeds 1024")
	}
	providers := make([]string, 0, len(groups))
	for provider := range groups {
		providers = append(providers, provider)
	}
	sort.Strings(providers)
	scenario := Scenario{Name: "provider-observed-agent-trace", Turns: len(records)}
	report := baseReport(BasisObserved, scenario, target, QualityTask)
	report.Scenario.TokenBasis = "provider-reported cache counters over caller-declared eligible input tokens"
	for _, provider := range providers {
		report.Providers = append(report.Providers, evaluateObservedProvider(provider, groups[provider], target))
	}
	report.Overall = aggregateProviders(report.Providers, target)
	if report.Overall.GatePassed {
		report.Status = "pass"
	}
	report.EvidenceLimitations = []string{
		"population completeness is not proven unless EvaluateObservedAgainstTrace is used",

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Normalize the Provider field (trim, lowercase, canonical name mapping) before evaluation
2. Audit distinct provider values in your records with a quick histogram and fix the source of the explosion
3. If genuinely measuring >1024 providers, shard the records and run EvaluateObserved per shard

Example fix

// before
report, err := cachebench.EvaluateObserved(rawRecords, target) // provider holds IDs

// after
for i := range rawRecords {
    rawRecords[i].Provider = canonicalProvider(rawRecords[i].Provider)
}
report, err := cachebench.EvaluateObserved(rawRecords, target)

distinct := map[string]bool{}
for _, r := range records {
    distinct[strings.ToLower(strings.TrimSpace(r.Provider))] = true
}
if len(distinct) > 1024 {
    return fmt.Errorf("%d distinct provider values; normalize provider field before evaluation", len(distinct))
}

func withinObservedProviderCap(rs []ObservationRecord) bool {
    m := map[string]bool{}
    for _, r := range rs { m[strings.ToLower(strings.TrimSpace(r.Provider))] = true }
    return len(m) <= 1024
}

When it happens

Trigger: Passing records whose Provider values contain >1024 distinct strings — e.g. provider polluted with model names, request IDs, or case/whitespace variants that survive even after the ToLower/TrimSpace normalization done at grouping.

Common situations: Log ingestion mapping the wrong column into Provider; mixed casings and trailing whitespace from multiple exporter versions; synthetic load tests with random provider tags.

Related errors

• cachebench: invalid observation read limits
• cachebench: observations exceed record limit %d
• cachebench: provider population exceeds 1024
• cachebench: no observation records
• cachebench: replay token summary overflow

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/d515d079832f6128. Report an issue: GitHub.