gastownhall/beads · error

timeout waiting for proxy on %s; safe child kill failed: %w

Error message

timeout waiting for proxy on %s; safe child kill failed: %w

What it means

Returned by spawnAndHandoff when the caller-provided deadline passes and the proxy is still not ready, and then the safe kill of the child fails. It layers a process-management error on top of the startup timeout, meaning an orphan child may remain.

Source

Thrown at internal/storage/dbproxy/proxy/endpoint.go:402

				return Endpoint{}, fmt.Errorf(
					"proxy child exited before becoming ready on explicitly configured port %d (see %s): %w",
					opts.Port, opts.LogFilePath, childErr,
				)
			}
			return Endpoint{}, fmt.Errorf(
				"proxy child exited before publishing its OS-assigned port (see %s): %w",
				opts.LogFilePath, childErr,
			)
		case <-hard.C:
			if err := killSpawnedChild(child); err != nil {
				return Endpoint{}, fmt.Errorf("hard timeout waiting for proxy on %s; safe child kill failed: %w", describeSpawnPort(opts.Port), err)
			}
			return Endpoint{}, fmt.Errorf("hard timeout (%s) waiting for proxy on %s", spawnReadyHardTimeout, describeSpawnPort(opts.Port))
		case <-poll.C:
		}
		if time.Now().After(deadline) {
			if err := killSpawnedChild(child); err != nil {
				return Endpoint{}, fmt.Errorf("timeout waiting for proxy on %s; safe child kill failed: %w", describeSpawnPort(opts.Port), err)
			}
			return Endpoint{}, fmt.Errorf("timeout waiting for proxy to become ready on %s", describeSpawnPort(opts.Port))
		}
	}
}

// describeSpawnPort renders a requested spawn port for wait/timeout
// messages: 0 is the default OS-assigned path, not a literal "port 0".
func describeSpawnPort(port int) string {
	if port == 0 {
		return "its OS-assigned port"
	}
	return fmt.Sprintf("port %d", port)
}

type spawnedProxyChild struct {
	cmd    *exec.Cmd
	done   <-chan error

View on GitHub (pinned to 71377f2769)

Solutions

1. Look for an orphaned db-proxy-child (pgrep) and clean it up so it doesn't hold the lock or port.
2. Increase the deadline passed to GetCreateDatabaseProxyServerEndpoint to accommodate cold-start time.
3. Diagnose why the kill failed (permissions, namespaces) before retrying to avoid accumulating zombies.
4. Retry the start after cleanup; the next attempt should adopt or spawn cleanly.

Example fix

// before: fixed short deadline on slow storage
ep, err := GetCreateDatabaseProxyServerEndpoint(root, opts) // deadline := time.Now().Add(2*time.Second)
// after: deadline scaled to environment
ep, err := GetCreateDatabaseProxyServerEndpoint(root, opts) // deadline := time.Now().Add(30*time.Second) on NFS/large DBs

ep, err := GetCreateDatabaseProxyServerEndpoint(rootDir, opts)
if err != nil && strings.Contains(err.Error(), "timeout waiting for proxy on") {
    // child kill also failed; ensure no orphan remains before retry
    exec.Command("pkill", "-f", "db-proxy-child --root "+rootDir).Run()
    ep, err = GetCreateDatabaseProxyServerEndpoint(rootDir, opts)
}

When it happens

Trigger: spawnAndHandoff's poll loop passes time.Now().After(deadline) (the caller's deadline passed to GetCreateDatabaseProxyServerEndpoint) with the child still unready, and killSpawnedChild errors — e.g. the process vanished between check and signal or signal permission was denied.

Common situations: Tight caller deadline on a slow cold start; parent and child in different containers/UIDs blocking the kill; PID-namespace mismatch causing ESRCH; test environments with aggressive process reaping.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• hard timeout waiting for proxy on %s; safe child kill failed
• timeout waiting for proxy to become ready on %s
• hard timeout (%s) waiting for proxy on %s
• errIdleTimeout
• dolt sql-server exited before listener became ready

AI-assisted analysis of gastownhall/beads@71377f2769 (2026-08-30). Data as JSON: /api/errors/1db32dba116b0609. Report an issue: GitHub.