{"record":{"id":"d515d079832f6128","repo":"JuliusBrussee/caveman","slug":"cachebench-provider-population-exceeds-1024-d515d0","errorCode":null,"errorMessage":"cachebench: provider population exceeds 1024","messagePattern":"cachebench: provider population exceeds 1024","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cacheengine/cachebench/observed.go","lineNumber":167,"sourceCode":"\n// EvaluateObserved evaluates supplied observations without completeness claim.\nfunc EvaluateObserved(records []ObservationRecord, target Target) (Report, error) {\n\tif err := validateTarget(target); err != nil {\n\t\treturn Report{}, err\n\t}\n\tif len(records) == 0 {\n\t\treturn Report{}, errors.New(\"cachebench: no observation records\")\n\t}\n\tif err := validateObservationRecords(records); err != nil {\n\t\treturn Report{}, err\n\t}\n\tgroups := map[string][]ObservationRecord{}\n\tfor _, record := range records {\n\t\tprovider := strings.ToLower(strings.TrimSpace(record.Provider))\n\t\tgroups[provider] = append(groups[provider], record)\n\t}\n\tif len(groups) > 1024 {\n\t\treturn Report{}, errors.New(\"cachebench: provider population exceeds 1024\")\n\t}\n\tproviders := make([]string, 0, len(groups))\n\tfor provider := range groups {\n\t\tproviders = append(providers, provider)\n\t}\n\tsort.Strings(providers)\n\tscenario := Scenario{Name: \"provider-observed-agent-trace\", Turns: len(records)}\n\treport := baseReport(BasisObserved, scenario, target, QualityTask)\n\treport.Scenario.TokenBasis = \"provider-reported cache counters over caller-declared eligible input tokens\"\n\tfor _, provider := range providers {\n\t\treport.Providers = append(report.Providers, evaluateObservedProvider(provider, groups[provider], target))\n\t}\n\treport.Overall = aggregateProviders(report.Providers, target)\n\tif report.Overall.GatePassed {\n\t\treport.Status = \"pass\"\n\t}\n\treport.EvidenceLimitations = []string{\n\t\t\"population completeness is not proven unless EvaluateObservedAgainstTrace is used\",","sourceCodeStart":149,"sourceCodeEnd":185,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/cacheengine/cachebench/observed.go#L149-L185","documentation":"EvaluateObserved groups records by lowercased trimmed Provider field and caps distinct providers at 1024, mirroring the corpus path's population bound. More than 1024 distinct provider strings indicates dirty data (unnormalized names, per-request IDs in the field) rather than a real provider set.","triggerScenarios":"Passing records whose Provider values contain >1024 distinct strings — e.g. provider polluted with model names, request IDs, or case/whitespace variants that survive even after the ToLower/TrimSpace normalization done at grouping.","commonSituations":"Log ingestion mapping the wrong column into Provider; mixed casings and trailing whitespace from multiple exporter versions; synthetic load tests with random provider tags.","solutions":["Normalize the Provider field (trim, lowercase, canonical name mapping) before evaluation","Audit distinct provider values in your records with a quick histogram and fix the source of the explosion","If genuinely measuring >1024 providers, shard the records and run EvaluateObserved per shard"],"exampleFix":"// before\nreport, err := cachebench.EvaluateObserved(rawRecords, target) // provider holds IDs\n\n// after\nfor i := range rawRecords {\n    rawRecords[i].Provider = canonicalProvider(rawRecords[i].Provider)\n}\nreport, err := cachebench.EvaluateObserved(rawRecords, target)","handlingStrategy":"validation","validationCode":"distinct := map[string]bool{}\nfor _, r := range records {\n    distinct[strings.ToLower(strings.TrimSpace(r.Provider))] = true\n}\nif len(distinct) > 1024 {\n    return fmt.Errorf(\"%d distinct provider values; normalize provider field before evaluation\", len(distinct))\n}","typeGuard":"func withinObservedProviderCap(rs []ObservationRecord) bool {\n    m := map[string]bool{}\n    for _, r := range rs { m[strings.ToLower(strings.TrimSpace(r.Provider))] = true }\n    return len(m) <= 1024\n}","tryCatchPattern":null,"preventionTips":["Canonicalize provider names (lowercase, trim, alias map) at ingestion time","Histogram distinct Provider values before evaluation to catch polluted fields","Never put request IDs or model names in the Provider field"],"tags":["go","observations","limits","data-hygiene"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}