JuliusBrussee/caveman · error

migrate sqlite data_ref

Error message

migrate sqlite data_ref %q: %w

What it means

This error wraps failure of ensureDataRefColumn, the migration step that adds the data_ref column (run inside RetryOnBusy) during store open. On failure the db is closed and open aborts. It indicates the ALTER TABLE-style migration for data_ref could not complete.

Solutions

1. Close other connections/processes to the db and retry
2. Verify write permission on the db file and directory
3. Unwrap the error chain to see the exact SQL failure and address it (e.g. disk full, corrupted page)
4. Restore the database from backup if the schema pages are corrupted

Example fix

// before
sudo systemctl start app # second instance, db locked
// after
systemctl stop app-old && systemctl start app # single writer only

if err := unix.Access(dbPath, unix.W_OK); err != nil {
    return fmt.Errorf("db %s not writable for migration: %w", dbPath, err)
}

store, err := ccr.Open(dbPath)
if err != nil && strings.Contains(err.Error(), "migrate sqlite data_ref") {
    // wait out the lock holder, then retry once
    time.Sleep(time.Second)
    store, err = ccr.Open(dbPath)
}

When it happens

Trigger: ensureDataRefColumn's queries fail or stay SQLITE_BUSY beyond retries; database file read-only; concurrent process blocking the migration.

Common situations: Version upgrade from a schema predating data_ref while the db is locked by another instance; filesystem quota/permission issues preventing writes.

Related errors

• migrate sqlite metadata
• migrate memories db
• migrate sqlite
• migrate sqlite
• broken supersession history at

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/5826d733b4490467. Report an issue: GitHub.

Appendix: source

Thrown at engine/ccr/store_sqlite.go:225

		// sidecars there would report a perfectly good database as replaced.
		mode, err := sqliteJournalMode(db)
		if err != nil {
			closeSQLiteAfterOpenFailure(db, canonicalPath)
			return nil, fmt.Errorf("read sqlite journal mode: %w", err)
		}
		walJournal = strings.EqualFold(mode, "wal")
	}
	if err := RetryOnBusy(func() error { _, e := db.Exec(schema); return e }); err != nil {
		closeSQLiteAfterOpenFailure(db, canonicalPath)
		return nil, fmt.Errorf("migrate sqlite %q: %w", canonicalPath, err)
	}
	if err := RetryOnBusy(func() error { return ensureMetadataColumn(db) }); err != nil {
		closeSQLiteAfterOpenFailure(db, canonicalPath)
		return nil, fmt.Errorf("migrate sqlite metadata %q: %w", canonicalPath, err)
	}
	if err := RetryOnBusy(func() error { return ensureDataRefColumn(db) }); err != nil {
		_ = db.Close()
		return nil, fmt.Errorf("migrate sqlite data_ref %q: %w", canonicalPath, err)
	}
	if err := configureStorageBudget(db, maxBytes); err != nil {
		closeSQLiteAfterOpenFailure(db, canonicalPath)
		return nil, fmt.Errorf("configure sqlite storage budget %q: %w", canonicalPath, err)
	}
	opened, err := inspectSQLiteGeneration(canonicalPath)
	if err != nil {
		closeSQLiteAfterOpenFailure(db, canonicalPath)
		return nil, err
	}
	if !before.sameExisting(opened) || (walJournal && (opened[1] == nil || opened[2] == nil)) {
		closeSQLiteAfterOpenFailure(db, canonicalPath)
		return nil, ErrStorageChanged
	}
	// The schema/budget queries have opened the WAL. Capture the resulting
	// journals before final permission checks, so losing a newly-created
	// journal is distinguishable from normal creation during SQLite open.
	before = opened

View on GitHub (pinned to 3ee70a1026)