What causes "store recovery for oversized memory %s: %w" in JuliusBrussee/caveman?

Inspect the wrapped %w error for the underlying SQLite failure.

store recovery for oversized memory

Error message

store recovery for oversized memory %s: %w

What it means

When a single recalled memory exceeds the token budget on its own, Recall falls back to a head-truncated 'cavemem-head' representation and must persist recovery metadata (the Hit with compressed text and a handle) into the store. If that storage write fails, the error is wrapped as "store recovery for oversized memory %s", naming the offending memory ID. Recall fails rather than silently returning an unrecoverable hit.

Solutions

Inspect the wrapped %w error for the underlying SQLite failure.
Ensure the store is open and writable (not closed, file not read-only) when calling Recall.
Resolve DB lock contention from concurrent writers before recalling.
Preemptively supersede or delete the oversized memory so the recovery path is not needed.

Example fix

// before
hits, err := store.Recall(ctx, q, mem.RecallOptions{TokenBudget: 100})
// after (verify store health and check wrapped error)
if err != nil {
    var id string
    if strings.Contains(err.Error(), "store recovery for oversized memory") {
        // extract ID, inspect/repair that memory, then retry
    }
    return err
}
hits, err := store.Recall(ctx, q, mem.RecallOptions{TokenBudget: 100})

Defensive patterns

Strategy: try-catch

Try / catch

hits, err := store.Recall(ctx, q, opts)
if err != nil && strings.Contains(err.Error(), "store recovery for oversized memory") {
    // DB write failed during head-truncation; check store health and retry
}

Prevention

Keep the store writable while Recall runs (don't Close early).
Supersede very large memories proactively so the recovery path isn't hit.
Set a TokenBudget large enough that common hits aren't oversized.

When it happens

Trigger: Recall returns a hit whose single memory's TokensBefore exceeds the budget, and the subsequent store write of the recovery Hit record fails (DB closed, locked, corrupted, disk full).

Common situations: Recalling a very large memory with a small TokenBudget while the SQLite file is read-only or the database handle was closed; also seen when a concurrent writer holds the write lock.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/e7ef2d1c6e45b18b. Report an issue: GitHub.

Appendix: source

Thrown at mem/store.go:554

// headHit builds a single recall hit for a memory whose compressed form exceeds
// the entire token budget: it truncates the compressed text to a head that fits
// and guarantees a CCR handle to the byte-exact original. When the engine passed
// the payload through without compressing (so it left no handle), the original
// is stored here — a head drops the tail, so the dropped detail must stay
// recoverable, upholding cavemem's reversible invariant.
func (s *Store) headHit(m Memory, compressedText []byte, handle string, score float64, budget int) (Hit, error) {
	head := truncateToTokens(compressedText, budget)
	if handle == "" {
		h, err := s.ccr.Put(ccr.Recovery{
			ContentType:  "text",
			Compressor:   "cavemem-head",
			TokensBefore: tokenCounter.Count([]byte(m.Text)),
			TokensAfter:  tokenCounter.Count(head),
			Original:     []byte(m.Text),
		})
		if err != nil {
			return Hit{}, fmt.Errorf("store recovery for oversized memory %s: %w", m.ID, err)
		}
		handle = h
	}
	return Hit{
		ID:             m.ID,
		Text:           string(head),
		Score:          score,
		TokensAdded:    tokenCounter.Count(head),
		Basis:          engine.BasisInferred,
		RecoveryHandle: handle,
	}, nil
}

// truncateToTokens returns the longest byte prefix of b whose inferred token
// count does not exceed budget, ending on a UTF-8 rune boundary. Recovery of the
// dropped tail is byte-exact through the hit's CCR handle.
func truncateToTokens(b []byte, budget int) []byte {
	if budget <= 0 {

View on GitHub (pinned to 3ee70a1026)