{"record":{"id":"678faf361191f663","repo":"JuliusBrussee/caveman","slug":"cachebench-observation-population-d-does-not-mat","errorCode":null,"errorMessage":"cachebench: observation population %d does not match trace population %d","messagePattern":"cachebench: observation population (.+?) does not match trace population (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cacheengine/cachebench/observed.go","lineNumber":38,"sourceCode":"\treturn ObservationRecord{\n\t\tSchema: ObservationSchema, RequestID: requestID, RequestBodySHA256: bodyDigest(originalRequestBody),\n\t\tProviderEvidenceSHA256: bodyDigest(providerEvidence),\n\t\tProvider:               provider, Epoch: epoch,\n\t\tEligibleInputTokens: eligibleInputTokens, CacheEligible: cacheEligible, Applied: result.Applied,\n\t\tEngineDecision: result.Decision, EngineReason: result.Reason, ProfileID: result.Profile.ID,\n\t\tAttribution: result.Profile.Attribution, OptimizerIDs: append([]string(nil), result.OptimizerIDs...),\n\t\tQualityPassed: verification.Passed, QualityVerifier: verification.Verifier,\n\t\tQualityEvidenceSHA256: bodyDigest(verification.Evidence), Usage: append(json.RawMessage(nil), rawUsage...),\n\t}\n}\n\n// EvaluateObservedAgainstTrace requires exact request population and body joins.\nfunc EvaluateObservedAgainstTrace(records []ObservationRecord, trace []TraceRecord, target Target) (Report, error) {\n\tif err := validateObservationRecords(records); err != nil {\n\t\treturn Report{}, err\n\t}\n\tif len(records) != len(trace) {\n\t\treturn Report{}, fmt.Errorf(\"cachebench: observation population %d does not match trace population %d\", len(records), len(trace))\n\t}\n\tbyID := make(map[string]TraceRecord, len(trace))\n\tfor index, request := range trace {\n\t\tif request.Schema != TraceSchema && request.Schema != TraceSchemaV2 || strings.TrimSpace(request.RequestID) == \"\" {\n\t\t\treturn Report{}, fmt.Errorf(\"cachebench: trace request %d has invalid schema or request_id\", index)\n\t\t}\n\t\tif _, exists := byID[request.RequestID]; exists {\n\t\t\treturn Report{}, fmt.Errorf(\"cachebench: duplicate trace request %q\", request.RequestID)\n\t\t}\n\t\tif strings.TrimSpace(request.Provider) == \"\" || strings.TrimSpace(request.Epoch) == \"\" || !validUniqueJSONObject(request.Body) || request.BodySHA256 == \"\" || request.BodySHA256 != bodyDigest(request.Body) {\n\t\t\treturn Report{}, fmt.Errorf(\"cachebench: trace request %q has invalid identity, body, or digest\", request.RequestID)\n\t\t}\n\t\tif request.StableSegmentCount < 0 || request.StableSegmentCount > len(request.Prefix) || !validPrefix(request.Prefix) {\n\t\t\treturn Report{}, fmt.Errorf(\"cachebench: trace request %q has invalid prefix\", request.RequestID)\n\t\t}\n\t\tbyID[request.RequestID] = request\n\t}\n\tfor _, record := range records {","sourceCodeStart":20,"sourceCodeEnd":56,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/cacheengine/cachebench/observed.go#L20-L56","documentation":"EvaluateObservedAgainstTrace requires the observation records and the request trace to cover exactly the same set of requests, so it first compares population sizes. A count mismatch means the join cannot be exact and the evaluation is refused rather than run on partial data.","triggerScenarios":"Calling EvaluateObservedAgainstTrace(records, trace, target) with len(records) != len(trace) — e.g. observations filtered or deduplicated independently of the trace, or a trace regenerated after the observation run.","commonSituations":"Replaying only part of a trace but keeping all observations (or vice versa); retry logic that appends duplicate observations; comparing observations from epoch A against a trace from epoch B.","solutions":["Regenerate the trace and observations from the same replay run so populations match by construction.","Filter both sides by the same request-ID set before evaluation.","Check for dropped observation lines (scanner errors, MaxRecords caps) that shrank the records slice.","Verify you are passing the trace file that corresponds to the observation file's epoch."],"exampleFix":"// before\nreport, err := EvaluateObservedAgainstTrace(records, traceA, target)\n// after\nreport, err := EvaluateObservedAgainstTrace(records, traceForSameRun, target)","handlingStrategy":"validation","validationCode":"func populationMatches(records []ObservationRecord, trace []TraceRecord) error {\n\tif len(records) != len(trace) {\n\t\treturn fmt.Errorf(\"population mismatch: %d observations vs %d trace\", len(records), len(trace))\n\t}\n\tids := map[string]bool{}\n\tfor _, t := range trace {\n\t\tids[t.RequestID] = true\n\t}\n\tfor _, r := range records {\n\t\tif !ids[r.RequestID] {\n\t\t\treturn fmt.Errorf(\"observation %q absent from trace\", r.RequestID)\n\t\t}\n\t}\n\treturn nil\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always generate trace and observations in the same replay run.","Never filter one side without applying the same filter to the other.","Store epoch/run IDs in filenames so mismatches are visible before loading."],"tags":["go","cachebench","evaluation","data-integrity"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}