{"record":{"id":"f95db148c0737673","repo":"block/buzz","slug":"database-migration-failed-e","errorCode":null,"errorMessage":"Database migration failed: {e}","messagePattern":"Database migration failed: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"critical","filePath":"crates/buzz-relay/src/main.rs","lineNumber":278,"sourceCode":"        error!(\"Failed to connect to Postgres: {e}\");\n        anyhow::anyhow!(\"DB connection failed: {e}\")\n    })?;\n    if db.has_read_pool() {\n        info!(\"Postgres connected (writer + lazy read replica pool)\");\n        // Reader-down at boot must not crash or block the relay; this warn-only\n        // ping is the sole boot-time visibility that the replica is unreachable\n        // (the lazy pool with min_connections=0 dials nothing until first use).\n        db.spawn_read_pool_boot_ping();\n    } else {\n        info!(\"Postgres connected\");\n    }\n\n    let auto_migrate =\n        buzz_auto_migrate_enabled(std::env::var(\"BUZZ_AUTO_MIGRATE\").ok().as_deref());\n    if auto_migrate {\n        db.migrate().await.map_err(|e| {\n            error!(\"Failed to run database migrations: {e}\");\n            anyhow::anyhow!(\"Database migration failed: {e}\")\n        })?;\n        info!(\"Database migrations complete\");\n    } else {\n        info!(\"Skipping database migrations because BUZZ_AUTO_MIGRATE is not enabled\");\n    }\n\n    if let Err(e) = db.ensure_future_partitions(3).await {\n        error!(\"Failed to ensure partitions: {e}\");\n    }\n\n    db.validate_deletion_serving_catalog().await.map_err(|e| {\n        error!(\"Community deletion serving-fence validation failed: {e}\");\n        anyhow::anyhow!(\"Community deletion serving fence is unsafe: {e}\")\n    })?;\n    info!(\"Community deletion serving fences verified\");\n\n    // Freshness fence probe: cursor pages route to the replica only for\n    // history the probe has verified as fully replayed. Deliberately AFTER","sourceCodeStart":260,"sourceCodeEnd":296,"githubUrl":"https://github.com/block/buzz/blob/ef2aa1ae38fadcc0bc22b8bf6ed96b35933146be/crates/buzz-relay/src/main.rs#L260-L296","documentation":"Fatal startup error: BUZZ_AUTO_MIGRATE is enabled and db.migrate() failed while applying the SQL migrations from migrations/. The {e} detail identifies the failing migration and the Postgres error; startup aborts rather than serve on a partially migrated schema.","triggerScenarios":"A migration's SQL fails on the live database: permission denied (e.g. CREATE EXTENSION as a non-superuser), lock timeout because another relay instance is migrating concurrently, unique/constraint violation from drifted data, or a schema object a previous manual change already created.","commonSituations":"Two relay replicas booting simultaneously against the same DB; a DB user with insufficient DDL rights; environments where migrations were applied by hand and now conflict; upgrading across many releases in one jump.","solutions":["Read the {e} detail to identify the failing migration file and SQL error code","If it is a lock/concurrency issue, let one instance finish (run one relay with BUZZ_AUTO_MIGRATE, others off) and restart","Grant the migrating user the needed privileges or run the failing migration manually with psql to surface the exact error","If data drift caused a constraint failure, repair or restore from snapshot, then re-run with a single migrating instance"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"# Dry-run migrations against a staging copy of prod before deploy:\n# pg_dump prod | psql staging && BUZZ_AUTO_MIGRATE=true ./buzz-relay --check || true\n# Inspect for failures before promoting the release.","typeGuard":null,"tryCatchPattern":"// Run exactly one migrating instance; on failure capture the migration id from\n// the error string, stop the deploy, and roll back the relay image — do not\n// let other replicas boot against the partially migrated DB.","preventionTips":["Migrate with a single relay instance; run others with BUZZ_AUTO_MIGRATE off until it finishes","Back up / snapshot the database before every migrating deploy","Grant the relay DB user only the DDL privileges migrations actually need"],"tags":["postgres","migration","startup","buzz-relay"],"backgroundTag":"database-migration-failed","analyzedSha":"ef2aa1ae38fadcc0bc22b8bf6ed96b35933146be","analyzedAt":"2026-08-20T04:38:24.874Z","contentChangedAt":"2026-08-20T04:38:24.874Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}