hard timeout waiting for proxy on %s; safe child kill failed

Error message

hard timeout waiting for proxy on %s; safe child kill failed: %w

What it means

Returned by spawnAndHandoff when the hard readiness timer (spawnReadyHardTimeout) fires and the subsequent attempt to safely kill the child process fails. It combines a startup timeout with a process-management failure, so both problems need attention.

Source

Thrown at internal/storage/dbproxy/proxy/endpoint.go:395

			if errors.As(childErr, &exitErr) && exitErr.ExitCode() == LockHeldExitCode {
				return Endpoint{}, fmt.Errorf(
					"proxy child lost the proxy.lock spawn race for %s: %w",
					rootDir, childErr,
				)
			}
			if opts.Port != 0 {
				return Endpoint{}, fmt.Errorf(
					"proxy child exited before becoming ready on explicitly configured port %d (see %s): %w",
					opts.Port, opts.LogFilePath, childErr,
				)
			}
			return Endpoint{}, fmt.Errorf(
				"proxy child exited before publishing its OS-assigned port (see %s): %w",
				opts.LogFilePath, childErr,
			)
		case <-hard.C:
			if err := killSpawnedChild(child); err != nil {
				return Endpoint{}, fmt.Errorf("hard timeout waiting for proxy on %s; safe child kill failed: %w", describeSpawnPort(opts.Port), err)
			}
			return Endpoint{}, fmt.Errorf("hard timeout (%s) waiting for proxy on %s", spawnReadyHardTimeout, describeSpawnPort(opts.Port))
		case <-poll.C:
		}
		if time.Now().After(deadline) {
			if err := killSpawnedChild(child); err != nil {
				return Endpoint{}, fmt.Errorf("timeout waiting for proxy on %s; safe child kill failed: %w", describeSpawnPort(opts.Port), err)
			}
			return Endpoint{}, fmt.Errorf("timeout waiting for proxy to become ready on %s", describeSpawnPort(opts.Port))
		}
	}
}

// describeSpawnPort renders a requested spawn port for wait/timeout
// messages: 0 is the default OS-assigned path, not a literal "port 0".
func describeSpawnPort(port int) string {
	if port == 0 {
		return "its OS-assigned port"

View on GitHub (pinned to 71377f2769)

Solutions

Investigate why killSpawnedChild failed (permissions, PID namespace, signal restrictions) — a zombie child may still be running and holding the lock/port.
Check the child log and ps/pgrep for a lingering db-proxy-child process; kill it manually if orphaned.
Free up system resources or increase spawnReadyHardTimeout if startup is legitimately slow.
Re-run the start in a cleaner environment (same user, same PID namespace) if container/permission quirks caused the kill failure.

Example fix

// before: ignoring a failed hard-timeout kill leaves orphan children
ep, err := GetCreateDatabaseProxyServerEndpoint(root, opts)
// after: on failure, sweep lingering children before retrying
if err != nil && strings.Contains(err.Error(), "safe child kill failed") {
    exec.Command("pkill", "-f", "db-proxy-child --root "+root).Run()
    ep, err = GetCreateDatabaseProxyServerEndpoint(root, opts)
}

Defensive patterns

Strategy: try-catch

Try / catch

ep, err := GetCreateDatabaseProxyServerEndpoint(rootDir, opts)
if err != nil && strings.Contains(err.Error(), "safe child kill failed") {
    // a child may be orphaned; sweep it before retrying
    exec.Command("pkill", "-f", "db-proxy-child --root "+rootDir).Run()
    time.Sleep(250 * time.Millisecond)
    ep, err = GetCreateDatabaseProxyServerEndpoint(rootDir, opts)
}

Prevention

Run the parent and child under the same UID so the kill signal is permitted
Avoid PID-namespace splitting between caller and proxy (same container/host)
Watch for lingering db-proxy-child processes after failures
Raise timeouts if the host is slow rather than letting the hard timer fire

When it happens

Trigger: The spawned proxy child neither becomes ready, exits, nor hits the caller deadline before spawnReadyHardTimeout elapses, and killSpawnedChild returns an error (e.g. process already reaped, permission denied sending the signal, or handle invalid).

Common situations: Extremely slow disk or database init exceeding the hard timeout; system under heavy load starving the child; permission issues (different uid, container restrictions) preventing the parent from killing the child; PID reuse/namespace quirks in containers.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of gastownhall/beads@71377f2769 (2026-08-30). Data as JSON: /api/errors/37bde28e75072ca0. Report an issue: GitHub.