{"record":{"id":"f95db148c0737673","repo":"block/buzz","slug":"database-migration-failed-e","errorCode":null,"errorMessage":"Database migration failed: {e}","messagePattern":"Database migration failed: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"critical","filePath":"crates/buzz-relay/src/main.rs","lineNumber":193,"sourceCode":"        error!(\"Failed to connect to Postgres: {e}\");\n        anyhow::anyhow!(\"DB connection failed: {e}\")\n    })?;\n    if db.has_read_pool() {\n        info!(\"Postgres connected (writer + lazy read replica pool)\");\n        // Reader-down at boot must not crash or block the relay; this warn-only\n        // ping is the sole boot-time visibility that the replica is unreachable\n        // (the lazy pool with min_connections=0 dials nothing until first use).\n        db.spawn_read_pool_boot_ping();\n    } else {\n        info!(\"Postgres connected\");\n    }\n\n    let auto_migrate =\n        buzz_auto_migrate_enabled(std::env::var(\"BUZZ_AUTO_MIGRATE\").ok().as_deref());\n    if auto_migrate {\n        db.migrate().await.map_err(|e| {\n            error!(\"Failed to run database migrations: {e}\");\n            anyhow::anyhow!(\"Database migration failed: {e}\")\n        })?;\n        info!(\"Database migrations complete\");\n    } else {\n        info!(\"Skipping database migrations because BUZZ_AUTO_MIGRATE is not enabled\");\n    }\n\n    if let Err(e) = db.ensure_future_partitions(3).await {\n        error!(\"Failed to ensure partitions: {e}\");\n    }\n\n    db.validate_deletion_serving_catalog().await.map_err(|e| {\n        error!(\"Community deletion serving-fence validation failed: {e}\");\n        anyhow::anyhow!(\"Community deletion serving fence is unsafe: {e}\")\n    })?;\n    info!(\"Community deletion serving fences verified\");\n\n    // Freshness fence probe: cursor pages route to the replica only for\n    // history the probe has verified as fully replayed. Deliberately AFTER","sourceCodeStart":175,"sourceCodeEnd":211,"githubUrl":"https://github.com/block/buzz/blob/f956e6fe06a76e50cbd8fba1a162482e752e7f1a/crates/buzz-relay/src/main.rs#L175-L211","documentation":"Raised when db.migrate() fails during startup with BUZZ_AUTO_MIGRATE enabled — the SQL migrations under migrations/ are applied on boot. Any failure (permissions, SQL error against a drifted schema, concurrent migrator) aborts startup with the underlying error appended.","triggerScenarios":"The DB user lacks DDL rights; two relay replicas boot and migrate simultaneously; a new migration hits a manually-edited or drifted schema; a restored backup has a partial migration history.","commonSituations":"Rolling/k8s deploys where old and new pods start together; RDS/CloudSQL users granted only DML; DBAs applying migrations manually out of order; test databases forked long ago.","solutions":["Read the migration number in {e} and open the failing file in migrations/","Grant DDL privileges or apply the pending migrations manually with psql as an admin","Run migrations from a single init container/job, then start relay pods with BUZZ_AUTO_MIGRATE disabled","If migration history is dirty/drifted, reconcile schema_migrations or restore from a known-good backup"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"# Verify every migration file's version is recorded before boot.\nlatest=$(ls migrations | grep -oE '^[0-9]+' | sort -n | tail -1)\napplied=$(psql \"$DATABASE_URL\" -tAc \"select max(version) from _sqlx_migrations\")\n[ \"$latest\" = \"$(echo $applied | tr -d ' ')\" ] || { echo \"pending migrations ($applied < $latest)\"; exit 1; }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Migrate from a single init container or job; run relay replicas with BUZZ_AUTO_MIGRATE off","Grant the runtime user DML only and migrate with a separate admin user","Rehearse migrations on a staging snapshot of production before deploy","Back up the database before any deploy that includes new migrations"],"tags":["postgres","migrations","startup","sqlx"],"backgroundTag":"database-migration-failed","analyzedSha":"f956e6fe06a76e50cbd8fba1a162482e752e7f1a","analyzedAt":"2026-08-16T22:11:40.750Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}