block/buzz · critical · anyhow::Error

Database migration failed: {e}

Error message

Database migration failed: {e}

What it means

Raised when db.migrate() fails during startup with BUZZ_AUTO_MIGRATE enabled — the SQL migrations under migrations/ are applied on boot. Any failure (permissions, SQL error against a drifted schema, concurrent migrator) aborts startup with the underlying error appended.

Source

Thrown at crates/buzz-relay/src/main.rs:193

        error!("Failed to connect to Postgres: {e}");
        anyhow::anyhow!("DB connection failed: {e}")
    })?;
    if db.has_read_pool() {
        info!("Postgres connected (writer + lazy read replica pool)");
        // Reader-down at boot must not crash or block the relay; this warn-only
        // ping is the sole boot-time visibility that the replica is unreachable
        // (the lazy pool with min_connections=0 dials nothing until first use).
        db.spawn_read_pool_boot_ping();
    } else {
        info!("Postgres connected");
    }

    let auto_migrate =
        buzz_auto_migrate_enabled(std::env::var("BUZZ_AUTO_MIGRATE").ok().as_deref());
    if auto_migrate {
        db.migrate().await.map_err(|e| {
            error!("Failed to run database migrations: {e}");
            anyhow::anyhow!("Database migration failed: {e}")
        })?;
        info!("Database migrations complete");
    } else {
        info!("Skipping database migrations because BUZZ_AUTO_MIGRATE is not enabled");
    }

    if let Err(e) = db.ensure_future_partitions(3).await {
        error!("Failed to ensure partitions: {e}");
    }

    db.validate_deletion_serving_catalog().await.map_err(|e| {
        error!("Community deletion serving-fence validation failed: {e}");
        anyhow::anyhow!("Community deletion serving fence is unsafe: {e}")
    })?;
    info!("Community deletion serving fences verified");

    // Freshness fence probe: cursor pages route to the replica only for
    // history the probe has verified as fully replayed. Deliberately AFTER

View on GitHub (pinned to f956e6fe06)

Solutions

1. Read the migration number in {e} and open the failing file in migrations/
2. Grant DDL privileges or apply the pending migrations manually with psql as an admin
3. Run migrations from a single init container/job, then start relay pods with BUZZ_AUTO_MIGRATE disabled
4. If migration history is dirty/drifted, reconcile schema_migrations or restore from a known-good backup

# Verify every migration file's version is recorded before boot.
latest=$(ls migrations | grep -oE '^[0-9]+' | sort -n | tail -1)
applied=$(psql "$DATABASE_URL" -tAc "select max(version) from _sqlx_migrations")
[ "$latest" = "$(echo $applied | tr -d ' ')" ] || { echo "pending migrations ($applied < $latest)"; exit 1; }

When it happens

Trigger: The DB user lacks DDL rights; two relay replicas boot and migrate simultaneously; a new migration hits a manually-edited or drifted schema; a restored backup has a partial migration history.

Common situations: Rolling/k8s deploys where old and new pods start together; RDS/CloudSQL users granted only DML; DBAs applying migrations manually out of order; test databases forked long ago.

Related errors

• DB connection failed: {e}
• Community deletion serving fence is unsafe: {e}
• Failed to ensure deployment community (required when BUZZ_RE
• Failed to backfill pubkey_allowlist (required when BUZZ_REQU
• Failed to bootstrap relay owner (required when BUZZ_REQUIRE_

AI-assisted analysis of block/buzz@f956e6fe06 (2026-08-16). Data as JSON: /api/errors/f95db148c0737673. Report an issue: GitHub.