block/buzz · critical · anyhow::Error

Database migration failed

Error message

Database migration failed: {e}

What it means

Fatal startup error: BUZZ_AUTO_MIGRATE is enabled and db.migrate() failed while applying the SQL migrations from migrations/. The {e} detail identifies the failing migration and the Postgres error; startup aborts rather than serve on a partially migrated schema.

Solutions

  1. Read the {e} detail to identify the failing migration file and SQL error code
  2. If it is a lock/concurrency issue, let one instance finish (run one relay with BUZZ_AUTO_MIGRATE, others off) and restart
  3. Grant the migrating user the needed privileges or run the failing migration manually with psql to surface the exact error
  4. If data drift caused a constraint failure, repair or restore from snapshot, then re-run with a single migrating instance
Defensive patterns

Strategy: try-catch

Validate before calling

# Dry-run migrations against a staging copy of prod before deploy:
# pg_dump prod | psql staging && BUZZ_AUTO_MIGRATE=true ./buzz-relay --check || true
# Inspect for failures before promoting the release.

Try / catch

// Run exactly one migrating instance; on failure capture the migration id from
// the error string, stop the deploy, and roll back the relay image — do not
// let other replicas boot against the partially migrated DB.

Prevention

When it happens

Trigger: A migration's SQL fails on the live database: permission denied (e.g. CREATE EXTENSION as a non-superuser), lock timeout because another relay instance is migrating concurrently, unique/constraint violation from drifted data, or a schema object a previous manual change already created.

Common situations: Two relay replicas booting simultaneously against the same DB; a DB user with insufficient DDL rights; environments where migrations were applied by hand and now conflict; upgrading across many releases in one jump.

Related errors

AI-assisted analysis of block/buzz@ef2aa1ae38 (2026-08-20). Data as JSON: /api/errors/f95db148c0737673. Report an issue: GitHub.

Appendix: source

Thrown at crates/buzz-relay/src/main.rs:278

        error!("Failed to connect to Postgres: {e}");
        anyhow::anyhow!("DB connection failed: {e}")
    })?;
    if db.has_read_pool() {
        info!("Postgres connected (writer + lazy read replica pool)");
        // Reader-down at boot must not crash or block the relay; this warn-only
        // ping is the sole boot-time visibility that the replica is unreachable
        // (the lazy pool with min_connections=0 dials nothing until first use).
        db.spawn_read_pool_boot_ping();
    } else {
        info!("Postgres connected");
    }

    let auto_migrate =
        buzz_auto_migrate_enabled(std::env::var("BUZZ_AUTO_MIGRATE").ok().as_deref());
    if auto_migrate {
        db.migrate().await.map_err(|e| {
            error!("Failed to run database migrations: {e}");
            anyhow::anyhow!("Database migration failed: {e}")
        })?;
        info!("Database migrations complete");
    } else {
        info!("Skipping database migrations because BUZZ_AUTO_MIGRATE is not enabled");
    }

    if let Err(e) = db.ensure_future_partitions(3).await {
        error!("Failed to ensure partitions: {e}");
    }

    db.validate_deletion_serving_catalog().await.map_err(|e| {
        error!("Community deletion serving-fence validation failed: {e}");
        anyhow::anyhow!("Community deletion serving fence is unsafe: {e}")
    })?;
    info!("Community deletion serving fences verified");

    // Freshness fence probe: cursor pages route to the replica only for
    // history the probe has verified as fully replayed. Deliberately AFTER

View on GitHub (pinned to ef2aa1ae38)