ccr: recovery storage changed; restart the process and…

Error message

ccr: recovery storage changed; restart the process and restore missing recovery files

What it means

CCR's sentinel ErrStorageChanged: the recovery database on disk no longer matches the open connection (generation/file inspection failed via inspectSQLiteGeneration/checkGeneration). The store fails closed, telling the caller to restart the process and restore missing recovery files rather than continue on unverified storage.

Solutions

Restart the process so the store reopens storage matching the current on-disk generation.
Restore the missing/moved recovery database files to the configured path.
Stop external processes (cleaners, sync tools) from deleting or moving the store directory.
Pin a stable, non-symlinked path for the recovery DB in configuration.

Example fix

// before
store, _ := ccr.Open(cfg) // path under volatile /tmp
defer store.Close()

// after
// use a persistent dir and monitor ErrStorageChanged
store, _ := ccr.Open(cfgWithPersistentPath)
if errors.Is(err, ccr.ErrStorageChanged) {
    return restartAndRestore(cfgWithPersistentPath, backup)
}

Defensive patterns

Strategy: try-catch

Validate before calling

if _, err := os.Stat(storePath); errors.Is(err, os.ErrNotExist) { return restoreBackup(storePath) }

Try / catch

if errors.Is(err, ccr.ErrStorageChanged) {
    log.Fatalf("ccr storage changed: restart process and restore %s", storePath)
}

Prevention

Place the recovery DB on persistent, non-ephemeral storage.
Exclude the store directory from cleaners/sync/tmp-reaping.
Use a real directory, not a symlink, for the DB path.
Keep verified backups to restore after a generation mismatch.

When it happens

Trigger: During Put or generation checks, inspectSQLiteGeneration detects: the database parent dir no longer resolves to itself (symlink/path changed), DB files deleted or replaced mid-operation, or the parent path became unverifiable (os.ErrNotExist).

Common situations: External cleanup jobs or tmpfs reaping removed the DB file; another process recreated/moved the recovery database; running in a container with an ephemeral filesystem wiping the store dir; symlinked paths retargeted.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/03fccc9f45fe6796. Report an issue: GitHub.

Appendix: source

Thrown at engine/ccr/store.go:38

	"errors"
	"fmt"
	"slices"
	"strings"
	"time"
)

// ErrNotFound is returned by Get when a handle is unknown. The store never
// guesses a recovery — an unknown handle is an explicit miss.
var ErrNotFound = errors.New("ccr: recovery handle not found")

// ErrBudgetExceeded means a new recovery was refused before publishing lossy
// bytes because the local store's configured payload budget would be exceeded.
// Existing handles remain intact and retrievable; callers must pass through.
var ErrBudgetExceeded = errors.New("ccr: storage budget exceeded")

// ErrStorageChanged means the recovery database no longer matches the open
// connection. Callers must preserve the original input and report the error.
var ErrStorageChanged = errors.New("ccr: recovery storage changed; restart the process and restore missing recovery files")

// ObjectType is a closed typed-working-memory enum. Unknown values fail closed:
// adapters may preserve unknown native payloads outside CCR, but may not invent
// retrieval semantics for them.
type ObjectType string

const (
	ObjectFileObservation      ObjectType = "FileObservation"
	ObjectSearchResult         ObjectType = "SearchResult"
	ObjectCommandResult        ObjectType = "CommandResult"
	ObjectTestResult           ObjectType = "TestResult"
	ObjectBuildResult          ObjectType = "BuildResult"
	ObjectDiffSnapshot         ObjectType = "DiffSnapshot"
	ObjectTaskContract         ObjectType = "TaskContract"
	ObjectTaskDecision         ObjectType = "TaskDecision"
	ObjectExecutionState       ObjectType = "ExecutionState"
	ObjectDocumentationExcerpt ObjectType = "DocumentationExcerpt"
	ObjectBrowserSnapshot      ObjectType = "BrowserSnapshot"

View on GitHub (pinned to 3ee70a1026)