neondatabase/neon · error

{} with pid {} did not stop in {:?} seconds

Error message

{} with pid {} did not stop in {:?} seconds

What it means

stop_process waits up to STOP_RETRIES iterations (STOP_RETRY_TIMEOUT total) for the process to disappear after SIGTERM/SIGKILL, polling process_has_stopped. If the pid still exists when the budget is exhausted, it bails with this timeout. Any error from the has-stopped probe is returned earlier as its own error.

Source

Thrown at control_plane/src/background_process.rs:247

                if retries == NOTICE_AFTER_RETRIES {
                    // The process is taking a long time to start up. Keep waiting, but
                    // print a message
                    print!("\n{process_name} has not stopped yet, continuing to wait");
                }
                if retries % DOT_EVERY_RETRIES == 0 {
                    print!(".");
                    io::stdout().flush().unwrap();
                }
                thread::sleep(RETRY_INTERVAL);
            }
            Err(e) => {
                println!("{process_name} with pid {pid} failed to stop: {e:#}");
                return Err(e);
            }
        }
    }
    println!();
    anyhow::bail!(format!(
        "{} with pid {} did not stop in {:?} seconds",
        process_name, pid, STOP_RETRY_TIMEOUT
    ));
}

fn fill_rust_env_vars(cmd: &mut Command) -> &mut Command {
    // If RUST_BACKTRACE is set, pass it through. But if it's not set, default
    // to RUST_BACKTRACE=1.
    let backtrace_setting = std::env::var_os("RUST_BACKTRACE");
    let backtrace_setting = backtrace_setting
        .as_deref()
        .unwrap_or_else(|| OsStr::new("1"));

    let mut filled_cmd = cmd.env_clear().env("RUST_BACKTRACE", backtrace_setting);

    // Pass through these environment variables to the command
    for var in [
        "LLVM_PROFILE_FILE",

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Retry the stop with immediate=true so SIGKILL is used instead of SIGTERM (`neon_local endpoint stop --immediate` or the equivalent call).
2. Check process state: `ps -o stat= -p <pid>` — D state means it's blocked on I/O, wait for disk/network to recover; Z means reap its parent.
3. For Postgres, reduce shutdown work beforehand (checkpoint_timeout / faster disks) or use fast/immediate stop modes via pg_ctl.
4. If a wrapper (supervisor, shell script) isn't reaping the child, fix that wrapper — a zombie pid never disappears.
5. After the process is confirmed dead manually, remove the stale pid file if nothing recreated it.

Example fix

// before
stop_process(false, process_name, &pid_file)?; // "did not stop in ... seconds"

// after
if stop_process(false, process_name, &pid_file).is_err() {
    // escalate: SIGKILL instead of SIGTERM
    stop_process(true, process_name, &pid_file)?;
}

// pre-escalation check: is the process killable at all?
if matches!(kill(pid, None), Ok(_)) && immediate == false {
    // expect possible SIGTERM resistance; budget for one escalation
}

if stop_process(false, process_name, &pid_file).is_err() {
    // graceful stop timed out -> escalate to SIGKILL
    stop_process(true, process_name, &pid_file)?;
}

When it happens

Trigger: Calling stop_process when the process ignores or delays handling SIGTERM: Postgres performing a long shutdown checkpoint or waiting on safekeeper sync, a hung binary, or a process stuck in uninterruptible I/O (D state). Also when SIGKILL couldn't be delivered because the process is a zombie whose parent hasn't reaped it.

Common situations: Stopping a compute with a large dirty buffer pool on slow disks; stopping while majority of safekeepers is unreachable so sync-safekeepers hangs; Docker-in-Docker or VM environments where SIGKILL delivery to zombies is delayed; CI machines under heavy load stretching checkpoint time past STOP_RETRY_TIMEOUT.

Related errors

• {} did not start+pass status checks within {:?} seconds
• Failed to send signal to {process_name} with pid {pid}: {e}
• `datadir` must be a directory when calling this function: {d
• process failed to start: {e}
• Failed to send signal to process with pid {pid}: {err}

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/da3ed0bda2afb479. Report an issue: GitHub.