JuliusBrussee/caveman · error

cachebench: invalid corpus

Error message

cachebench: invalid corpus

What it means

RunCorpus validates the corpus shape before doing any work: it must contain at least one row, a non-empty SHA256 digest, and a non-empty trimmed Metadata.Name. Any of these missing yields 'cachebench: invalid corpus', since a benchmark over an empty or unidentifiable corpus is meaningless.

Source

Thrown at cacheengine/cachebench/corpus.go:445

	hash := sha256.New()
	encoder := json.NewEncoder(hash)
	encoder.SetEscapeHTML(false)
	for _, row := range rows {
		_ = encoder.Encode(row)
	}
	return hex.EncodeToString(hash.Sum(nil))
}

// RunCorpus evaluates one validated public corpus across provider compilers.
func RunCorpus(ctx context.Context, engine *cacheengine.Engine, corpus AgentCorpus, providers []ProviderConfig, target Target) (Report, error) {
	if engine == nil {
		return Report{}, errors.New("cachebench: nil cache engine")
	}
	if err := validateTarget(target); err != nil {
		return Report{}, err
	}
	if len(corpus.Rows) == 0 || corpus.SHA256 == "" || strings.TrimSpace(corpus.Metadata.Name) == "" {
		return Report{}, errors.New("cachebench: invalid corpus")
	}
	if len(providers) == 0 {
		return Report{}, errors.New("cachebench: no providers")
	}
	if len(providers) > 1024 {
		return Report{}, errors.New("cachebench: provider population exceeds 1024")
	}
	counter := tokens.Default()
	cachedCounter := &corpusTokenCounter{Counter: counter, counts: map[[sha256.Size]byte]int{}}
	summary, err := analyzeCorpus(corpus, cachedCounter)
	if err != nil {
		return Report{}, err
	}
	scenario := Scenario{Name: "public-agent-corpus", Turns: len(corpus.Rows), Step: time.Second, AssumedTTL: 5 * time.Minute}
	report := baseReport(BasisCorpusSimulated, scenario, target, QualityEquivalence)
	report.Corpus = &summary
	report.Scenario.TokenBasis = counter.Name() + " estimate over normalized OpenAI messages"
	report.Scenario.Step = "corpus pre_gap"

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Verify the corpus source file actually decoded (len(Rows) > 0) before benchmarking
2. Populate corpus.SHA256 with the hex sha256 of the canonical row encoding and set Metadata.Name to a real identifier
3. Log corpus.Rows/Sessions counts right after load to catch empty inputs early

Example fix

// before
corpus := AgentCorpus{} // decoded from empty file
report, err := cachebench.RunCorpus(ctx, engine, corpus, providers, target)

// after
if len(corpus.Rows) == 0 || corpus.SHA256 == "" || strings.TrimSpace(corpus.Metadata.Name) == "" {
    return fmt.Errorf("corpus not ready: rows=%d sha=%q", len(corpus.Rows), corpus.SHA256)
}
report, err := cachebench.RunCorpus(ctx, engine, corpus, providers, target)

func corpusReady(c AgentCorpus) bool {
    return len(c.Rows) > 0 && c.SHA256 != "" && strings.TrimSpace(c.Metadata.Name) != ""
}

if !corpusReady(corpus) {
    return fmt.Errorf("corpus incomplete: rows=%d sha256_set=%v name=%q", len(corpus.Rows), corpus.SHA256 != "", corpus.Metadata.Name)
}

func isValidCorpus(c AgentCorpus) bool {
    return len(c.Rows) > 0 && c.SHA256 != "" && strings.TrimSpace(c.Metadata.Name) != ""
}

if err := runCorpusWrapper(); err != nil {
    if strings.Contains(err.Error(), "invalid corpus") {
        // re-check corpus fields and reload/regenerate the corpus file
    }
}

When it happens

Trigger: Passing an AgentCorpus with empty Rows, empty SHA256, or whitespace-only Metadata.Name to RunCorpus. Typical after a load/parse step produced an empty struct or the digest step was skipped when building the corpus.

Common situations: Corpus loaded from a JSON file that failed silently or was empty; SHA256 computed into a different field or never assigned; Metadata left zero-valued in a hand-built corpus.

Related errors

• cachebench: nil corpus reader
• message exceeds byte limit
• tool message requires tool_call_id
• cachebench: no providers
• cachebench: corpus has no sessions

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/8f6f4bfa5c258f9a. Report an issue: GitHub.