cachebench: observation population %d does not match trace p

Error message

cachebench: observation population %d does not match trace population %d

What it means

EvaluateObservedAgainstTrace requires the observation records and the request trace to cover exactly the same set of requests, so it first compares population sizes. A count mismatch means the join cannot be exact and the evaluation is refused rather than run on partial data.

Source

Thrown at cacheengine/cachebench/observed.go:38

	return ObservationRecord{
		Schema: ObservationSchema, RequestID: requestID, RequestBodySHA256: bodyDigest(originalRequestBody),
		ProviderEvidenceSHA256: bodyDigest(providerEvidence),
		Provider:               provider, Epoch: epoch,
		EligibleInputTokens: eligibleInputTokens, CacheEligible: cacheEligible, Applied: result.Applied,
		EngineDecision: result.Decision, EngineReason: result.Reason, ProfileID: result.Profile.ID,
		Attribution: result.Profile.Attribution, OptimizerIDs: append([]string(nil), result.OptimizerIDs...),
		QualityPassed: verification.Passed, QualityVerifier: verification.Verifier,
		QualityEvidenceSHA256: bodyDigest(verification.Evidence), Usage: append(json.RawMessage(nil), rawUsage...),
	}
}

// EvaluateObservedAgainstTrace requires exact request population and body joins.
func EvaluateObservedAgainstTrace(records []ObservationRecord, trace []TraceRecord, target Target) (Report, error) {
	if err := validateObservationRecords(records); err != nil {
		return Report{}, err
	}
	if len(records) != len(trace) {
		return Report{}, fmt.Errorf("cachebench: observation population %d does not match trace population %d", len(records), len(trace))
	}
	byID := make(map[string]TraceRecord, len(trace))
	for index, request := range trace {
		if request.Schema != TraceSchema && request.Schema != TraceSchemaV2 || strings.TrimSpace(request.RequestID) == "" {
			return Report{}, fmt.Errorf("cachebench: trace request %d has invalid schema or request_id", index)
		}
		if _, exists := byID[request.RequestID]; exists {
			return Report{}, fmt.Errorf("cachebench: duplicate trace request %q", request.RequestID)
		}
		if strings.TrimSpace(request.Provider) == "" || strings.TrimSpace(request.Epoch) == "" || !validUniqueJSONObject(request.Body) || request.BodySHA256 == "" || request.BodySHA256 != bodyDigest(request.Body) {
			return Report{}, fmt.Errorf("cachebench: trace request %q has invalid identity, body, or digest", request.RequestID)
		}
		if request.StableSegmentCount < 0 || request.StableSegmentCount > len(request.Prefix) || !validPrefix(request.Prefix) {
			return Report{}, fmt.Errorf("cachebench: trace request %q has invalid prefix", request.RequestID)
		}
		byID[request.RequestID] = request
	}
	for _, record := range records {

View on GitHub (pinned to 27d5a3981a)

Solutions

Regenerate the trace and observations from the same replay run so populations match by construction.
Filter both sides by the same request-ID set before evaluation.
Check for dropped observation lines (scanner errors, MaxRecords caps) that shrank the records slice.
Verify you are passing the trace file that corresponds to the observation file's epoch.

Example fix

// before
report, err := EvaluateObservedAgainstTrace(records, traceA, target)
// after
report, err := EvaluateObservedAgainstTrace(records, traceForSameRun, target)

Defensive patterns

Strategy: validation

Validate before calling

func populationMatches(records []ObservationRecord, trace []TraceRecord) error {
	if len(records) != len(trace) {
		return fmt.Errorf("population mismatch: %d observations vs %d trace", len(records), len(trace))
	}
	ids := map[string]bool{}
	for _, t := range trace {
		ids[t.RequestID] = true
	}
	for _, r := range records {
		if !ids[r.RequestID] {
			return fmt.Errorf("observation %q absent from trace", r.RequestID)
		}
	}
	return nil
}

Prevention

Always generate trace and observations in the same replay run.
Never filter one side without applying the same filter to the other.
Store epoch/run IDs in filenames so mismatches are visible before loading.

When it happens

Trigger: Calling EvaluateObservedAgainstTrace(records, trace, target) with len(records) != len(trace) — e.g. observations filtered or deduplicated independently of the trace, or a trace regenerated after the observation run.

Common situations: Replaying only part of a trace but keeping all observations (or vice versa); retry logic that appends duplicate observations; comparing observations from epoch A against a trace from epoch B.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/678faf361191f663. Report an issue: GitHub.