JuliusBrussee/caveman · error

ccr: recovery store is closed

Error message

ccr: recovery store is closed

What it means

The CCR recovery store refuses all operations after Close() has been called. checkGeneration runs on every withStore/Close path and short-circuits with this sentinel when s.closed is set, preventing use of a store whose recovery files may already have been released.

Solutions

  1. Ensure Close is the last operation: reorder code so no withStore call happens after Close.
  2. Audit goroutines holding a reference to the store and shut them down before Close (waitgroup/context cancellation).
  3. If the store must be reopened, construct a new Store instead of reusing the closed instance.
  4. Guard call sites with a closed check or run operations through a single owner that serializes Close.

Example fix

// before
store.Close()
record, err := store.WithStore(func(...) {...}) // panics-free but errors
// after
record, err := store.WithStore(func(...) {...})
if err != nil { ... }
store.Close()
Defensive patterns

Strategy: try-catch

Validate before calling

// track liveness yourself
if store.IsClosed() { return errors.New("store already closed") }

Try / catch

rec, err := store.WithStore(fn)
if err != nil {
    if strings.Contains(err.Error(), "store is closed") {
        // reopen or abort shutdown path
    }
    return err
}

Prevention

When it happens

Trigger: Any operation routed through withStore (reads/writes to the recovery store) or Close itself, executed after a prior call to Store.Close on the same *ccr.Store instance.

Common situations: Calling Close in a defer and then still using the store later in the function; a long-lived process that closed the store during shutdown but a lingering goroutine still writes decisions; double-Close; tests reusing a closed fixture store.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/e6d2c2e4e3e428e9. Report an issue: GitHub.

Appendix: source

Thrown at engine/ccr/store_generation.go:141

// before the first complete generation snapshot. Never close an opened disk
// connection on that error path: SQLite close can checkpoint its stale WAL into
// the current main file. The process must exit to release those descriptors.
func closeSQLiteAfterOpenFailure(db *sql.DB, path string) {
	if path == ":memory:" || db.Stats().OpenConnections == 0 {
		_ = db.Close()
	}
}

// checkGeneration runs with mu held. A detected generation change is terminal
// for this Store. A fresh process must open the restored/current database;
// pathname snapshots cannot establish which files a replacement connection
// actually opened, so automatically adopting a replacement is unsafe.
func (s *Store) checkGeneration() error {
	if s.quarantined != nil {
		return s.quarantined
	}
	if s.closed {
		return errors.New("ccr: recovery store is closed")
	}
	files, err := inspectSQLiteGeneration(s.path)
	if errors.Is(err, errStorageUnverifiable) {
		// "Could not look" is not "was replaced". A momentary stat/permission
		// failure — an antivirus lock on Windows, EIO on a network home, a
		// parent directory whose mode was loose for one instant — fails THIS
		// operation closed, but must not latch the terminal state below and
		// make already-stored recoveries unreadable for the rest of the process.
		return err
	}
	if err != nil {
		s.quarantined = err
		return s.quarantined
	}
	if s.db != nil && s.files.same(files) {
		return nil
	}
	// No SQLite calls, including Close, follow invalidation. The public driver

View on GitHub (pinned to 3ee70a1026)