What causes "timeout waiting for job manager to start" in wavetermdev/waveterm?

Check remote host load and I/O latency; retry under lighter load

timeout waiting for job manager to start

Error message

timeout waiting for job manager to start

What it means

RemoteStartJobCommand waits up to 5 seconds (context.WithTimeout) for the 'Wave-JobManagerStart' signal on the ready pipe. If the timeout fires first, the child is killed and this error is returned — the jobmanager started but never became ready.

Source

Thrown at pkg/wshrpc/wshremote/wshremote_job.go:245

		}
	}()

	timeoutCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
	defer cancel()

	log.Printf("RemoteStartJobCommand: waiting for start signal\n")
	select {
	case err := <-startCh:
		if err != nil {
			cmd.Process.Kill()
			log.Printf("RemoteStartJobCommand: error from start signal: %v\n", err)
			return nil, err
		}
		log.Printf("RemoteStartJobCommand: received start signal\n")
	case <-timeoutCtx.Done():
		cmd.Process.Kill()
		log.Printf("RemoteStartJobCommand: timeout waiting for start signal\n")
		return nil, fmt.Errorf("timeout waiting for job manager to start")
	}

	go func() {
		cmd.Wait()
	}()

	jobRouteId, cleanup, err := impl.connectToJobManager(ctx, data.JobId, data.MainServerJwtToken)
	if err != nil {
		return nil, err
	}

	combinedEnv := make(map[string]string)
	for k, v := range impl.InitialEnv {
		combinedEnv[k] = v
	}
	for k, v := range data.Env {
		combinedEnv[k] = v
	}

View on GitHub (pinned to a4447c1563)

Solutions

Check remote host load and I/O latency; retry under lighter load
Inspect stderr logs to find where the jobmanager is stalling
Update the wsh binary via 'wsh init' to rule out protocol mismatch
If startups are legitimately slow, increase the 5s timeout in RemoteStartJobCommand

Example fix

// before
timeoutCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
// after
timeoutCtx, cancel := context.WithTimeout(ctx, 30*time.Second)

Defensive patterns

Strategy: retry

Validate before calling

// check the remote can start wsh promptly
start := time.Now()
out, err := exec.Command(wshPath, "--version").CombinedOutput()
if err != nil || time.Since(start) > 3*time.Second {
    return fmt.Errorf("wsh slow or broken on remote (%.1fs): %v", time.Since(start).Seconds(), err)
}

Try / catch

rtn, err := server.RemoteStartJobCommand(ctx, data)
if err != nil {
    if strings.Contains(err.Error(), "timeout waiting for job manager") {
        // retry under lighter load or after increasing the startup timeout
    }
    return err
}

Prevention

Monitor remote host load; avoid job starts during heavy load
Raise the 5s startup timeout if hosts are legitimately slow
Keep wsh versions in sync so the child never blocks waiting on unknown input

When it happens

Trigger: The jobmanager process runs but hangs before emitting the start signal: slow startup (heavy load, cold filesystem), a hang waiting on network/auth setup, or a binary that never writes the signal to fd 3.

Common situations: Overloaded or swapping remote host where wsh startup exceeds 5s; hung connection handshake inside the child; version mismatch causing the child to wait for input that never comes.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of wavetermdev/waveterm@a4447c1563 (2026-09-01). Data as JSON: /api/errors/ca02db48284b47bc. Report an issue: GitHub.