gastownhall/beads · error

hard timeout (%s) waiting for proxy on %s

Error message

hard timeout (%s) waiting for proxy on %s

What it means

Returned by spawnAndHandoff when the hard readiness timer (spawnReadyHardTimeout) expires before the proxy child became ready, exited, or the caller's deadline passed — and the child was killed successfully. It reports the configured hard timeout value and the target port description.

Source

Thrown at internal/storage/dbproxy/proxy/endpoint.go:397

					"proxy child lost the proxy.lock spawn race for %s: %w",
					rootDir, childErr,
				)
			}
			if opts.Port != 0 {
				return Endpoint{}, fmt.Errorf(
					"proxy child exited before becoming ready on explicitly configured port %d (see %s): %w",
					opts.Port, opts.LogFilePath, childErr,
				)
			}
			return Endpoint{}, fmt.Errorf(
				"proxy child exited before publishing its OS-assigned port (see %s): %w",
				opts.LogFilePath, childErr,
			)
		case <-hard.C:
			if err := killSpawnedChild(child); err != nil {
				return Endpoint{}, fmt.Errorf("hard timeout waiting for proxy on %s; safe child kill failed: %w", describeSpawnPort(opts.Port), err)
			}
			return Endpoint{}, fmt.Errorf("hard timeout (%s) waiting for proxy on %s", spawnReadyHardTimeout, describeSpawnPort(opts.Port))
		case <-poll.C:
		}
		if time.Now().After(deadline) {
			if err := killSpawnedChild(child); err != nil {
				return Endpoint{}, fmt.Errorf("timeout waiting for proxy on %s; safe child kill failed: %w", describeSpawnPort(opts.Port), err)
			}
			return Endpoint{}, fmt.Errorf("timeout waiting for proxy to become ready on %s", describeSpawnPort(opts.Port))
		}
	}
}

// describeSpawnPort renders a requested spawn port for wait/timeout
// messages: 0 is the default OS-assigned path, not a literal "port 0".
func describeSpawnPort(port int) string {
	if port == 0 {
		return "its OS-assigned port"
	}
	return fmt.Sprintf("port %d", port)

View on GitHub (pinned to 71377f2769)

Solutions

  1. Check the child log at opts.LogFilePath and any partial discovery record for how far startup got.
  2. Retry once — transient slowness under load often resolves on a warm start.
  3. If startup is legitimately slow, increase the hard timeout constant (spawnReadyHardTimeout) or raise the caller deadline.
  4. Rule out environmental slowdowns (NFS latency, AV scanning, resource limits) on the host.

Example fix

// before: huge database cold start exceeds the fixed hard timeout
ep, err := GetCreateDatabaseProxyServerEndpoint(bigRoot, opts) // hard timeout fires
// after: pre-warm/start the backend ahead of time, or rely on an already-running proxy
if existing, derr := DiscoverDatabaseProxyServerEndpoint(bigRoot); derr == nil {
    ep = existing
} else {
    ep, err = GetCreateDatabaseProxyServerEndpoint(bigRoot, opts)
}
Defensive patterns

Strategy: retry

Validate before calling

// Estimate feasibility: ensure workspace database exists and disk isn't thrashing
if _, err := os.Stat(filepath.Join(rootDir, ".dolt")); err != nil {
    return fmt.Errorf("no dolt database at %s: %w", rootDir, err)
}

Try / catch

ep, err := GetCreateDatabaseProxyServerEndpoint(rootDir, opts)
if err != nil && strings.Contains(err.Error(), "hard timeout") {
    time.Sleep(time.Second) // let the killed child fully release the lock/port
    ep, err = GetCreateDatabaseProxyServerEndpoint(rootDir, opts)
}

Prevention

When it happens

Trigger: GetCreateDatabaseProxyServerEndpoint's child takes longer than spawnReadyHardTimeout to publish a dialable endpoint: slow backend start, blocked bind, wedged filesystem, or a child hanging before it can exit.

Common situations: Cold start on a very large Dolt database; NFS/slow workspace storage; host under heavy CPU load; a hung dolt subprocess inside the child; security software (AV) delaying process startup.

Understand the failure class

Related errors

AI-assisted analysis of gastownhall/beads@71377f2769 (2026-08-30). Data as JSON: /api/errors/33a36576c959bf22. Report an issue: GitHub.