What causes "failed to reconnect to job manager: %s" in wavetermdev/waveterm?

Read rtnData.Error (embedded in the message) for the remote's specific failure reason.

failed to reconnect to job manager: %s

Error message

failed to reconnect to job manager: %s

What it means

This error is returned when the RemoteReconnectToJobManagerCommand RPC completed but the remote side answered Success=false without JobManagerGone (jobcontroller.go:1145). The remote daemon attempted to reattach to the job manager and failed for a reason other than the process being gone (e.g. auth token mismatch, pid reuse check failed). The remote's error string is included verbatim.

Source

Thrown at pkg/jobcontroller/jobcontroller.go:1145

				job.JobManagerDoneReason = JobDoneReason_Gone
				updatedJob = job
			})
			if updateErr != nil {
				log.Printf("[job:%s] error updating job manager running status: %v", jobId, updateErr)
			} else {
				sendBlockJobStatusEventByJob(ctx, updatedJob)
			}
			telemetry.GoRecordTEventWrap(&telemetrydata.TEvent{
				Event: "job:done",
				Props: telemetrydata.TEventProps{
					JobDoneReason: JobDoneReason_Gone,
					JobKind:       job.JobKind,
				},
			})
			writeJobTerminationMessage(ctx, jobId, updatedJob, "[session gone]")
			return fmt.Errorf("job manager has exited: %s", rtnData.Error)
		}
		return fmt.Errorf("failed to reconnect to job manager: %s", rtnData.Error)
	}

	log.Printf("[job:%s] RemoteReconnectToJobManagerCommand succeeded, waiting for route", jobId)

	routeId := wshutil.MakeJobRouteId(jobId)
	waitCtx, cancelFn := context.WithTimeout(ctx, 2*time.Second)
	defer cancelFn()
	err = wshutil.DefaultRouter.WaitForRegister(waitCtx, routeId)
	if err != nil {
		return fmt.Errorf("route did not establish after successful reconnection: %w", err)
	}
	SetJobConnStatus(jobId, JobConnStatus_Connected)
	sendBlockJobStatusEventByJob(ctx, job)

	telemetry.GoRecordTEventWrap(&telemetrydata.TEvent{
		Event: "job:reconnect",
		Props: telemetrydata.TEventProps{
			JobKind: job.JobKind,

View on GitHub (pinned to a4447c1563)

Solutions

Read rtnData.Error (embedded in the message) for the remote's specific failure reason.
Retry the reconnect; transient remote-state issues may resolve on a second attempt.
If persistent, terminate the job and start a new one since reattachment keeps failing.
Ensure client and remote wsh versions are compatible.

Defensive patterns

Strategy: retry

Validate before calling

// ensure client and remote wsh versions match before reconnect

Try / catch

if err := jobcontroller.ReconnectJob(ctx, jobId, nil); err != nil && strings.Contains(err.Error(), "failed to reconnect to job manager") && !strings.Contains(err.Error(), "has exited") {
    // remote rejected: inspect embedded reason, optionally retry once
}

Prevention

Keep wsh versions in sync between client and remote
Retry once on transient remote-state errors, then give up
Log the embedded remote error string for diagnosis

When it happens

Trigger: rtnData.Success is false and rtnData.JobManagerGone is false — the remote explicitly rejected the reconnect with its own error message in rtnData.Error.

Common situations: 1) Stale job.JobAuthToken / pid metadata that no longer matches the running manager. 2) The remote daemon restarted and partially lost state about the job manager. 3) Version mismatch between client and remote wsh during the reconnect handshake.

Related errors

AI-assisted analysis of wavetermdev/waveterm@a4447c1563 (2026-09-01). Data as JSON: /api/errors/575388e56aec73a3. Report an issue: GitHub.