{"record":{"id":"9fce7825a226ec54","repo":"neondatabase/neon","slug":"failed-to-stop-storage-controller-database-err","errorCode":null,"errorMessage":"Failed to stop storage controller database: {err}","messagePattern":"Failed to stop storage controller database: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"control_plane/src/storage_controller.rs","lineNumber":747,"sourceCode":"            }\n        }\n\n        let pg_data_path = self.env.base_data_dir.join(\"storage_controller_db\");\n\n        println!(\"Stopping storage controller database...\");\n        let pg_stop_args = [\"-D\", &pg_data_path.to_string_lossy(), \"stop\"];\n        let stop_status = self.pg_ctl(pg_stop_args).await;\n        if !stop_status.success() {\n            match self.is_postgres_running().await {\n                Ok(false) => {\n                    println!(\"Storage controller database is already stopped\");\n                    return Ok(());\n                }\n                Ok(true) => {\n                    anyhow::bail!(\"Failed to stop storage controller database\");\n                }\n                Err(err) => {\n                    anyhow::bail!(\"Failed to stop storage controller database: {err}\");\n                }\n            }\n        }\n\n        Ok(())\n    }\n\n    async fn is_postgres_running(&self) -> anyhow::Result<bool> {\n        let pg_data_path = self.env.base_data_dir.join(\"storage_controller_db\");\n\n        let pg_status_args = [\"-D\", &pg_data_path.to_string_lossy(), \"status\"];\n        let status_exitcode = self.pg_ctl(pg_status_args).await;\n\n        // pg_ctl status returns this exit code if postgres is not running: in this case it is\n        // fine that stop failed.  Otherwise it is an error that stop failed.\n        const PG_STATUS_NOT_RUNNING: i32 = 3;\n        const PG_NO_DATA_DIR: i32 = 4;\n        const PG_STATUS_RUNNING: i32 = 0;","sourceCodeStart":729,"sourceCodeEnd":765,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/control_plane/src/storage_controller.rs#L729-L765","documentation":"Same stop path, but here the follow-up pg_ctl status check itself failed (unexpected exit code, no exit code, or spawn error), so storcon cannot even tell whether postgres is still running and reports the failed stop together with the status-check error.","triggerScenarios":"pg_ctl stop failed AND pg_ctl status returned an exit code outside {0,3,4} or was killed by a signal — e.g. permission problems on the data dir or a broken pg_ctl environment.","commonSituations":"Data dir ownership changed between start and stop, pg binaries moved/removed since start, filesystem errors on the volume holding storage_controller_db.","solutions":["Resolve the pg_ctl status failure first: run pg_ctl -D <base>/storage_controller_db status manually and check permissions and that pg_ctl still exists","Then retry the stop and reconcile any leftover postgres process"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"match pg_ctl([\"-D\", &dir, \"status\"]).await.code() {\n    Some(0) | Some(3) | Some(4) => { /* known state: proceed with stop logic */ }\n    other => { /* surface pg_ctl stderr / fix perms before interpreting stop failure */ }\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep start/stop under the same user and the same pg build","Verify the pg_ctl binary still exists before status calls","Capture pg_ctl stderr in logs for post-mortem"],"tags":["rust","storage-controller","postgres","pg-ctl","shutdown"],"backgroundTag":"postgres-stop-failed","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}