JuliusBrussee/caveman · error

cachebench: provider population exceeds 1024

Error message

cachebench: provider population exceeds 1024

What it means

RunCorpus caps the provider population at 1024 to bound compile time and report size. More than 1024 ProviderConfig entries is almost certainly a configuration mistake (duplicate expansion, cartesian product of models) rather than a real benchmark matrix.

Source

Thrown at cacheengine/cachebench/corpus.go:451

	return hex.EncodeToString(hash.Sum(nil))
}

// RunCorpus evaluates one validated public corpus across provider compilers.
func RunCorpus(ctx context.Context, engine *cacheengine.Engine, corpus AgentCorpus, providers []ProviderConfig, target Target) (Report, error) {
	if engine == nil {
		return Report{}, errors.New("cachebench: nil cache engine")
	}
	if err := validateTarget(target); err != nil {
		return Report{}, err
	}
	if len(corpus.Rows) == 0 || corpus.SHA256 == "" || strings.TrimSpace(corpus.Metadata.Name) == "" {
		return Report{}, errors.New("cachebench: invalid corpus")
	}
	if len(providers) == 0 {
		return Report{}, errors.New("cachebench: no providers")
	}
	if len(providers) > 1024 {
		return Report{}, errors.New("cachebench: provider population exceeds 1024")
	}
	counter := tokens.Default()
	cachedCounter := &corpusTokenCounter{Counter: counter, counts: map[[sha256.Size]byte]int{}}
	summary, err := analyzeCorpus(corpus, cachedCounter)
	if err != nil {
		return Report{}, err
	}
	scenario := Scenario{Name: "public-agent-corpus", Turns: len(corpus.Rows), Step: time.Second, AssumedTTL: 5 * time.Minute}
	report := baseReport(BasisCorpusSimulated, scenario, target, QualityEquivalence)
	report.Corpus = &summary
	report.Scenario.TokenBasis = counter.Name() + " estimate over normalized OpenAI messages"
	report.Scenario.Step = "corpus pre_gap"
	report.Scenario.AssumedTTL = "provider profile TTL"
	for _, provider := range providers {
		trace, buildErr := buildCorpusTrace(provider, corpus, cachedCounter)
		if buildErr != nil {
			return Report{}, buildErr
		}

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Deduplicate and trim the provider list to the matrix you actually intend to measure
2. Bound generator loops (providers × models) and assert the product stays under 1024 before calling RunCorpus
3. If you genuinely need more, split into multiple RunCorpus invocations and merge reports yourself

Example fix

// before
providers := expandCartesian(allNames, allModels) // 2000 entries
report, err := cachebench.RunCorpus(ctx, engine, corpus, providers, target)

// after
providers := dedupeProviders(expandCartesian(allNames, allModels))
if len(providers) > 1024 {
    return fmt.Errorf("provider matrix too large: %d", len(providers))
}
report, err := cachebench.RunCorpus(ctx, engine, corpus, providers, target)

if len(providers) > 1024 {
    return fmt.Errorf("provider matrix of %d exceeds cap 1024; dedupe or shard the benchmark", len(providers))
}
report, err := cachebench.RunCorpus(ctx, engine, corpus, providers, target)

func withinProviderCap(ps []ProviderConfig) bool { return len(ps) <= 1024 }

When it happens

Trigger: Passing >1024 ProviderConfig entries — commonly from generating providers × models × variants programmatically, or reading a config where a provider stanza got duplicated by a templating loop.

Common situations: Script builds providers as cross-product of 100 names × 20 models; YAML anchor mis-use duplicating lists; unit test generating synthetic provider IDs without bounding.

Related errors

• row limit %d exceeded
• cachebench: no providers
• cachebench: invalid observation read limits
• cacheengine: stable prefix exceeds configured byte limit
• cacheengine: request exceeds configured byte limit

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/ecaa3a037d9d9b89. Report an issue: GitHub.