wavetermdev/waveterm · error

job manager has exited: %s

Error message

job manager has exited: %s

What it means

This error indicates the remote side explicitly reported that the job manager process is gone (rtnData.JobManagerGone true) during a reconnect attempt (jobcontroller.go:1143). The library marks the job as done with JobDoneReason_Gone, writes a '[session gone]' termination message, and returns this error — the job cannot be reattached because its managing process no longer exists on the remote host (e.g. host rebooted).

Source

Thrown at pkg/jobcontroller/jobcontroller.go:1143

			updateErr := wstore.DBUpdateFn(ctx, jobId, func(job *waveobj.Job) {
				job.JobManagerStatus = JobManagerStatus_Done
				job.JobManagerDoneReason = JobDoneReason_Gone
				updatedJob = job
			})
			if updateErr != nil {
				log.Printf("[job:%s] error updating job manager running status: %v", jobId, updateErr)
			} else {
				sendBlockJobStatusEventByJob(ctx, updatedJob)
			}
			telemetry.GoRecordTEventWrap(&telemetrydata.TEvent{
				Event: "job:done",
				Props: telemetrydata.TEventProps{
					JobDoneReason: JobDoneReason_Gone,
					JobKind:       job.JobKind,
				},
			})
			writeJobTerminationMessage(ctx, jobId, updatedJob, "[session gone]")
			return fmt.Errorf("job manager has exited: %s", rtnData.Error)
		}
		return fmt.Errorf("failed to reconnect to job manager: %s", rtnData.Error)
	}

	log.Printf("[job:%s] RemoteReconnectToJobManagerCommand succeeded, waiting for route", jobId)

	routeId := wshutil.MakeJobRouteId(jobId)
	waitCtx, cancelFn := context.WithTimeout(ctx, 2*time.Second)
	defer cancelFn()
	err = wshutil.DefaultRouter.WaitForRegister(waitCtx, routeId)
	if err != nil {
		return fmt.Errorf("route did not establish after successful reconnection: %w", err)
	}
	SetJobConnStatus(jobId, JobConnStatus_Connected)
	sendBlockJobStatusEventByJob(ctx, job)

	telemetry.GoRecordTEventWrap(&telemetrydata.TEvent{
		Event: "job:reconnect",

View on GitHub (pinned to a4447c1563)

Solutions

1. Treat the job as finished — reconnect is impossible; open a new job/terminal instead.
2. Surface the '[session gone]' message to the user (the library already writes it via writeJobTerminationMessage).
3. For long-running remote work, run jobs under a supervisor (tmux/systemd) on the remote host so they survive daemon restarts.
4. Clear/ignore the job record; the DB was already updated to JobManagerStatus_Done.

Example fix

// before
err := jobcontroller.ReconnectJob(ctx, jobId, nil) // fails: job manager gone
// after
if err := jobcontroller.ReconnectJob(ctx, jobId, nil); err != nil {
    log.Printf("job unrecoverable, creating new job: %v", err)
    // fall back to launching a fresh job/block
}

// no local check can predict remote process death; handle after the fact

if err := jobcontroller.ReconnectJob(ctx, jobId, nil); err != nil && strings.Contains(err.Error(), "job manager has exited") {
    // unrecoverable: create a new job / notify user session is gone
}

When it happens

Trigger: RemoteReconnectToJobManagerCommand succeeds as an RPC but returns Success=false with JobManagerGone=true — the remote wsh daemon looked for pid job.JobManagerPid (started at job.JobManagerStartTs) and found it exited.

Common situations: 1) The remote machine rebooted between sessions, killing the job manager. 2) The remote session (e.g. SSH-launched daemon) was cleaned up when the last connection closed. 3) The job manager process crashed or was OOM-killed on the remote host.

Related errors

• failed to reconnect to job manager: %s
• beginning of file
• procinfo: process not found
• integer overflow
• bind tags must be self closing

AI-assisted analysis of wavetermdev/waveterm@a4447c1563 (2026-09-01). Data as JSON: /api/errors/dc9023bc12aa2a70. Report an issue: GitHub.