What causes "failed to reconnect to job manager: %w" in wavetermdev/waveterm?

Retry ReconnectJob; the reconnectGroup deduplicates concurrent attempts.

failed to reconnect to job manager: %w

Error message

failed to reconnect to job manager: %w

What it means

This error is returned when the RPC call wshclient.RemoteReconnectToJobManagerCommand fails at the transport/RPC level while trying to reconnect a job manager over its connection (jobcontroller.go:1118). It wraps the underlying error from the RPC round-trip, which may be a route error, timeout (5000ms here), or the remote peer rejecting the call.

Source

Thrown at pkg/jobcontroller/jobcontroller.go:1118

	reconnectData := wshrpc.CommandRemoteReconnectToJobManagerData{
		JobId:              jobId,
		JobAuthToken:       job.JobAuthToken,
		MainServerJwtToken: jobAccessToken,
		JobManagerPid:      job.JobManagerPid,
		JobManagerStartTs:  job.JobManagerStartTs,
	}

	rpcOpts := &wshrpc.RpcOpts{
		Route:   wshutil.MakeConnectionRouteId(job.Connection),
		Timeout: 5000,
	}

	log.Printf("[job:%s] sending RemoteReconnectToJobManagerCommand to connection %s", jobId, job.Connection)
	rtnData, err := wshclient.RemoteReconnectToJobManagerCommand(bareRpc, reconnectData, rpcOpts)
	if err != nil {
		log.Printf("[job:%s] RemoteReconnectToJobManagerCommand failed: %v", jobId, err)
		return fmt.Errorf("failed to reconnect to job manager: %w", err)
	}

	if !rtnData.Success {
		log.Printf("[job:%s] RemoteReconnectToJobManagerCommand returned error: %s", jobId, rtnData.Error)
		if rtnData.JobManagerGone {
			var updatedJob *waveobj.Job
			updateErr := wstore.DBUpdateFn(ctx, jobId, func(job *waveobj.Job) {
				job.JobManagerStatus = JobManagerStatus_Done
				job.JobManagerDoneReason = JobDoneReason_Gone
				updatedJob = job
			})
			if updateErr != nil {
				log.Printf("[job:%s] error updating job manager running status: %v", jobId, updateErr)
			} else {
				sendBlockJobStatusEventByJob(ctx, updatedJob)
			}
			telemetry.GoRecordTEventWrap(&telemetrydata.TEvent{
				Event: "job:done",

View on GitHub (pinned to a4447c1563)

Solutions

Retry ReconnectJob; the reconnectGroup deduplicates concurrent attempts.
Verify the connection is still alive and reconnect it if needed, then retry.
Check logs for the underlying wrapped error (e.g. 'no route', 'timeout') to distinguish transport vs timeout.
Increase robustness by ensuring the remote wsh daemon is running on the target host.

Defensive patterns

Strategy: retry

Validate before calling

connected, _ := conncontroller.IsConnected(connName)
// only attempt RPC reconnect when conn is live

Try / catch

if err := jobcontroller.ReconnectJob(ctx, jobId, nil); err != nil && strings.Contains(err.Error(), "failed to reconnect to job manager") {
    // transient RPC failure: retry with backoff
}

Prevention

Retry reconnects with exponential backoff
Keep the connection alive during reconnect flows
Watch for RPC timeout errors distinguishing slow network from dead routes

When it happens

Trigger: RemoteReconnectToJobManagerCommand returns an error: the connection route (MakeConnectionRouteId(job.Connection)) has no responder, the RPC timed out after 5s, or the bare RPC client lost its link while the conn still reports connected.

Common situations: 1) Connection dropped between the IsConnected check and the RPC call (race). 2) Remote wsh daemon died so no route exists for the conn. 3) Slow/broken network causing the 5-second RPC timeout.

Related errors

AI-assisted analysis of wavetermdev/waveterm@a4447c1563 (2026-09-01). Data as JSON: /api/errors/3d56e3b8bffb8ab2. Report an issue: GitHub.