{"record":{"id":"da3ed0bda2afb479","repo":"neondatabase/neon","slug":"with-pid-did-not-stop-in-seconds","errorCode":null,"errorMessage":"{} with pid {} did not stop in {:?} seconds","messagePattern":"(.+?) with pid (.+?) did not stop in (.+?) seconds","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"control_plane/src/background_process.rs","lineNumber":247,"sourceCode":"                if retries == NOTICE_AFTER_RETRIES {\n                    // The process is taking a long time to start up. Keep waiting, but\n                    // print a message\n                    print!(\"\\n{process_name} has not stopped yet, continuing to wait\");\n                }\n                if retries % DOT_EVERY_RETRIES == 0 {\n                    print!(\".\");\n                    io::stdout().flush().unwrap();\n                }\n                thread::sleep(RETRY_INTERVAL);\n            }\n            Err(e) => {\n                println!(\"{process_name} with pid {pid} failed to stop: {e:#}\");\n                return Err(e);\n            }\n        }\n    }\n    println!();\n    anyhow::bail!(format!(\n        \"{} with pid {} did not stop in {:?} seconds\",\n        process_name, pid, STOP_RETRY_TIMEOUT\n    ));\n}\n\nfn fill_rust_env_vars(cmd: &mut Command) -> &mut Command {\n    // If RUST_BACKTRACE is set, pass it through. But if it's not set, default\n    // to RUST_BACKTRACE=1.\n    let backtrace_setting = std::env::var_os(\"RUST_BACKTRACE\");\n    let backtrace_setting = backtrace_setting\n        .as_deref()\n        .unwrap_or_else(|| OsStr::new(\"1\"));\n\n    let mut filled_cmd = cmd.env_clear().env(\"RUST_BACKTRACE\", backtrace_setting);\n\n    // Pass through these environment variables to the command\n    for var in [\n        \"LLVM_PROFILE_FILE\",","sourceCodeStart":229,"sourceCodeEnd":265,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/control_plane/src/background_process.rs#L229-L265","documentation":"stop_process waits up to STOP_RETRIES iterations (STOP_RETRY_TIMEOUT total) for the process to disappear after SIGTERM/SIGKILL, polling process_has_stopped. If the pid still exists when the budget is exhausted, it bails with this timeout. Any error from the has-stopped probe is returned earlier as its own error.","triggerScenarios":"Calling stop_process when the process ignores or delays handling SIGTERM: Postgres performing a long shutdown checkpoint or waiting on safekeeper sync, a hung binary, or a process stuck in uninterruptible I/O (D state). Also when SIGKILL couldn't be delivered because the process is a zombie whose parent hasn't reaped it.","commonSituations":"Stopping a compute with a large dirty buffer pool on slow disks; stopping while majority of safekeepers is unreachable so sync-safekeepers hangs; Docker-in-Docker or VM environments where SIGKILL delivery to zombies is delayed; CI machines under heavy load stretching checkpoint time past STOP_RETRY_TIMEOUT.","solutions":["Retry the stop with immediate=true so SIGKILL is used instead of SIGTERM (`neon_local endpoint stop --immediate` or the equivalent call).","Check process state: `ps -o stat= -p <pid>` — D state means it's blocked on I/O, wait for disk/network to recover; Z means reap its parent.","For Postgres, reduce shutdown work beforehand (checkpoint_timeout / faster disks) or use fast/immediate stop modes via pg_ctl.","If a wrapper (supervisor, shell script) isn't reaping the child, fix that wrapper — a zombie pid never disappears.","After the process is confirmed dead manually, remove the stale pid file if nothing recreated it."],"exampleFix":"// before\nstop_process(false, process_name, &pid_file)?; // \"did not stop in ... seconds\"\n\n// after\nif stop_process(false, process_name, &pid_file).is_err() {\n    // escalate: SIGKILL instead of SIGTERM\n    stop_process(true, process_name, &pid_file)?;\n}","handlingStrategy":"retry","validationCode":"// pre-escalation check: is the process killable at all?\nif matches!(kill(pid, None), Ok(_)) && immediate == false {\n    // expect possible SIGTERM resistance; budget for one escalation\n}","typeGuard":null,"tryCatchPattern":"if stop_process(false, process_name, &pid_file).is_err() {\n    // graceful stop timed out -> escalate to SIGKILL\n    stop_process(true, process_name, &pid_file)?;\n}","preventionTips":["Always provide an immediate/SIGKILL escalation path when wrapping stop_process.","Ensure safekeepers are reachable before stopping a primary compute, so sync work can't hang.","For Postgres, prefer fast/immediate stop modes in tests that don't need clean checkpoints.","Watch for D-state processes (blocked I/O) — no signal helps until the I/O completes."],"tags":["rust","neon","control-plane","background-process","shutdown","timeout","process-management"],"backgroundTag":"service-shutdown-timeout","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}