JuliusBrussee/caveman · error

cachebench: no replay evidence

Error message

cachebench: no replay evidence

What it means

SummarizeReplayEvidence aggregates ReplayEvidenceRecord values into a summary and requires at least one record. An empty slice cannot produce token totals, latency distributions, or pass counts, so it errors rather than returning a zero summary that could be mistaken for a real (all-zero) replay.

Source

Thrown at cacheengine/cachebench/replay.go:248

// ReplayEvidenceSummary aggregates exact validated evidence population.
type ReplayEvidenceSummary struct {
	Schema                            string                  `json:"schema"`
	Requests                          int                     `json:"requests"`
	Successful                        int                     `json:"successful"`
	Failed                            int                     `json:"failed"`
	QualityPassed                     int                     `json:"quality_passed"`
	InputTokens                       int64                   `json:"provider_total_input_tokens"`
	OutputTokens                      int64                   `json:"provider_output_tokens"`
	TimingFaithful                    bool                    `json:"timing_faithful"`
	InputBudgetClaimedProviderCounted bool                    `json:"input_budget_claimed_provider_counted"`
	Latency                           LatencyDistribution     `json:"latency"`
	Providers                         []ProviderReplaySummary `json:"providers"`
}

// SummarizeReplayEvidence validates and aggregates replay records.
func SummarizeReplayEvidence(records []ReplayEvidenceRecord) (ReplayEvidenceSummary, error) {
	if len(records) == 0 {
		return ReplayEvidenceSummary{}, errors.New("cachebench: no replay evidence")
	}
	summary := ReplayEvidenceSummary{
		Schema: ReplaySummarySchema, Requests: len(records),
		TimingFaithful: true, InputBudgetClaimedProviderCounted: true,
	}
	type accumulator struct {
		summary   ProviderReplaySummary
		latencies []int64
	}
	groups := map[string]*accumulator{}
	seen := map[string]bool{}
	latencies := make([]int64, 0, len(records))
	for index, record := range records {
		if err := validateReplayEvidence(record); err != nil {
			return ReplayEvidenceSummary{}, fmt.Errorf("cachebench: replay evidence %d: %w", index, err)
		}
		if seen[record.RequestID] {
			return ReplayEvidenceSummary{}, fmt.Errorf("cachebench: duplicate replay evidence request %q", record.RequestID)

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Check len(records) > 0 after replay and before summarization; treat zero as a replay failure, not a summary input
2. Verify the evidence JSONL was written and non-empty after the replay phase
3. Log replay counts per provider so an empty result is visible with context

Example fix

// before
summary, err := cachebench.SummarizeReplayEvidence(records) // empty

// after
if len(records) == 0 {
    return fmt.Errorf("replay produced no evidence; check replay filters and evidence output path")
}
summary, err := cachebench.SummarizeReplayEvidence(records)

if len(records) == 0 {
    return errors.New("replay produced no evidence; refusing to summarize")
}
summary, err := cachebench.SummarizeReplayEvidence(records)

func hasReplayEvidence(rs []ReplayEvidenceRecord) bool { return len(rs) > 0 }

if _, err := cachebench.SummarizeReplayEvidence(records); err != nil {
    if err.Error() == "cachebench: no replay evidence" {
        // check replay stage logs: filters, dry-run flags, evidence file path
    }
}

When it happens

Trigger: Calling SummarizeReplayEvidence with an empty records slice — typically the replay run produced no evidence (all requests filtered, JSONL empty, or the replay loop never executed) before summarization.

Common situations: Replay pipeline stage skipped due to dry-run flag but summarize still called; evidence file empty after a crashed replay; records filtered by request ID set that matched nothing.

Related errors

• cachebench: corpus has no sessions
• cachebench: nil corpus reader
• message exceeds byte limit
• tool message requires tool_call_id
• cachebench: invalid corpus

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/a2ec2af2295f3c5b. Report an issue: GitHub.