{"record":{"id":"ecaa3a037d9d9b89","repo":"JuliusBrussee/caveman","slug":"cachebench-provider-population-exceeds-1024","errorCode":null,"errorMessage":"cachebench: provider population exceeds 1024","messagePattern":"cachebench: provider population exceeds 1024","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cacheengine/cachebench/corpus.go","lineNumber":451,"sourceCode":"\treturn hex.EncodeToString(hash.Sum(nil))\n}\n\n// RunCorpus evaluates one validated public corpus across provider compilers.\nfunc RunCorpus(ctx context.Context, engine *cacheengine.Engine, corpus AgentCorpus, providers []ProviderConfig, target Target) (Report, error) {\n\tif engine == nil {\n\t\treturn Report{}, errors.New(\"cachebench: nil cache engine\")\n\t}\n\tif err := validateTarget(target); err != nil {\n\t\treturn Report{}, err\n\t}\n\tif len(corpus.Rows) == 0 || corpus.SHA256 == \"\" || strings.TrimSpace(corpus.Metadata.Name) == \"\" {\n\t\treturn Report{}, errors.New(\"cachebench: invalid corpus\")\n\t}\n\tif len(providers) == 0 {\n\t\treturn Report{}, errors.New(\"cachebench: no providers\")\n\t}\n\tif len(providers) > 1024 {\n\t\treturn Report{}, errors.New(\"cachebench: provider population exceeds 1024\")\n\t}\n\tcounter := tokens.Default()\n\tcachedCounter := &corpusTokenCounter{Counter: counter, counts: map[[sha256.Size]byte]int{}}\n\tsummary, err := analyzeCorpus(corpus, cachedCounter)\n\tif err != nil {\n\t\treturn Report{}, err\n\t}\n\tscenario := Scenario{Name: \"public-agent-corpus\", Turns: len(corpus.Rows), Step: time.Second, AssumedTTL: 5 * time.Minute}\n\treport := baseReport(BasisCorpusSimulated, scenario, target, QualityEquivalence)\n\treport.Corpus = &summary\n\treport.Scenario.TokenBasis = counter.Name() + \" estimate over normalized OpenAI messages\"\n\treport.Scenario.Step = \"corpus pre_gap\"\n\treport.Scenario.AssumedTTL = \"provider profile TTL\"\n\tfor _, provider := range providers {\n\t\ttrace, buildErr := buildCorpusTrace(provider, corpus, cachedCounter)\n\t\tif buildErr != nil {\n\t\t\treturn Report{}, buildErr\n\t\t}","sourceCodeStart":433,"sourceCodeEnd":469,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/cacheengine/cachebench/corpus.go#L433-L469","documentation":"RunCorpus caps the provider population at 1024 to bound compile time and report size. More than 1024 ProviderConfig entries is almost certainly a configuration mistake (duplicate expansion, cartesian product of models) rather than a real benchmark matrix.","triggerScenarios":"Passing >1024 ProviderConfig entries — commonly from generating providers × models × variants programmatically, or reading a config where a provider stanza got duplicated by a templating loop.","commonSituations":"Script builds providers as cross-product of 100 names × 20 models; YAML anchor mis-use duplicating lists; unit test generating synthetic provider IDs without bounding.","solutions":["Deduplicate and trim the provider list to the matrix you actually intend to measure","Bound generator loops (providers × models) and assert the product stays under 1024 before calling RunCorpus","If you genuinely need more, split into multiple RunCorpus invocations and merge reports yourself"],"exampleFix":"// before\nproviders := expandCartesian(allNames, allModels) // 2000 entries\nreport, err := cachebench.RunCorpus(ctx, engine, corpus, providers, target)\n\n// after\nproviders := dedupeProviders(expandCartesian(allNames, allModels))\nif len(providers) > 1024 {\n    return fmt.Errorf(\"provider matrix too large: %d\", len(providers))\n}\nreport, err := cachebench.RunCorpus(ctx, engine, corpus, providers, target)","handlingStrategy":"validation","validationCode":"if len(providers) > 1024 {\n    return fmt.Errorf(\"provider matrix of %d exceeds cap 1024; dedupe or shard the benchmark\", len(providers))\n}\nreport, err := cachebench.RunCorpus(ctx, engine, corpus, providers, target)","typeGuard":"func withinProviderCap(ps []ProviderConfig) bool { return len(ps) <= 1024 }","tryCatchPattern":null,"preventionTips":["Bound provider-generation loops and assert the product of names × models up front","Deduplicate provider configs before benchmarking","Split matrices larger than 1024 into multiple RunCorpus calls and merge reports"],"tags":["go","corpus","limits","configuration"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}