{"record":{"id":"dc9023bc12aa2a70","repo":"wavetermdev/waveterm","slug":"job-manager-has-exited-s","errorCode":null,"errorMessage":"job manager has exited: %s","messagePattern":"job manager has exited: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/jobcontroller/jobcontroller.go","lineNumber":1143,"sourceCode":"\t\t\tupdateErr := wstore.DBUpdateFn(ctx, jobId, func(job *waveobj.Job) {\n\t\t\t\tjob.JobManagerStatus = JobManagerStatus_Done\n\t\t\t\tjob.JobManagerDoneReason = JobDoneReason_Gone\n\t\t\t\tupdatedJob = job\n\t\t\t})\n\t\t\tif updateErr != nil {\n\t\t\t\tlog.Printf(\"[job:%s] error updating job manager running status: %v\", jobId, updateErr)\n\t\t\t} else {\n\t\t\t\tsendBlockJobStatusEventByJob(ctx, updatedJob)\n\t\t\t}\n\t\t\ttelemetry.GoRecordTEventWrap(&telemetrydata.TEvent{\n\t\t\t\tEvent: \"job:done\",\n\t\t\t\tProps: telemetrydata.TEventProps{\n\t\t\t\t\tJobDoneReason: JobDoneReason_Gone,\n\t\t\t\t\tJobKind:       job.JobKind,\n\t\t\t\t},\n\t\t\t})\n\t\t\twriteJobTerminationMessage(ctx, jobId, updatedJob, \"[session gone]\")\n\t\t\treturn fmt.Errorf(\"job manager has exited: %s\", rtnData.Error)\n\t\t}\n\t\treturn fmt.Errorf(\"failed to reconnect to job manager: %s\", rtnData.Error)\n\t}\n\n\tlog.Printf(\"[job:%s] RemoteReconnectToJobManagerCommand succeeded, waiting for route\", jobId)\n\n\trouteId := wshutil.MakeJobRouteId(jobId)\n\twaitCtx, cancelFn := context.WithTimeout(ctx, 2*time.Second)\n\tdefer cancelFn()\n\terr = wshutil.DefaultRouter.WaitForRegister(waitCtx, routeId)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"route did not establish after successful reconnection: %w\", err)\n\t}\n\tSetJobConnStatus(jobId, JobConnStatus_Connected)\n\tsendBlockJobStatusEventByJob(ctx, job)\n\n\ttelemetry.GoRecordTEventWrap(&telemetrydata.TEvent{\n\t\tEvent: \"job:reconnect\",","sourceCodeStart":1125,"sourceCodeEnd":1161,"githubUrl":"https://github.com/wavetermdev/waveterm/blob/a4447c1563b2df285ab89e76c82f91e1a1a49c1e/pkg/jobcontroller/jobcontroller.go#L1125-L1161","documentation":"This error indicates the remote side explicitly reported that the job manager process is gone (rtnData.JobManagerGone true) during a reconnect attempt (jobcontroller.go:1143). The library marks the job as done with JobDoneReason_Gone, writes a '[session gone]' termination message, and returns this error — the job cannot be reattached because its managing process no longer exists on the remote host (e.g. host rebooted).","triggerScenarios":"RemoteReconnectToJobManagerCommand succeeds as an RPC but returns Success=false with JobManagerGone=true — the remote wsh daemon looked for pid job.JobManagerPid (started at job.JobManagerStartTs) and found it exited.","commonSituations":"1) The remote machine rebooted between sessions, killing the job manager. 2) The remote session (e.g. SSH-launched daemon) was cleaned up when the last connection closed. 3) The job manager process crashed or was OOM-killed on the remote host.","solutions":["Treat the job as finished — reconnect is impossible; open a new job/terminal instead.","Surface the '[session gone]' message to the user (the library already writes it via writeJobTerminationMessage).","For long-running remote work, run jobs under a supervisor (tmux/systemd) on the remote host so they survive daemon restarts.","Clear/ignore the job record; the DB was already updated to JobManagerStatus_Done."],"exampleFix":"// before\nerr := jobcontroller.ReconnectJob(ctx, jobId, nil) // fails: job manager gone\n// after\nif err := jobcontroller.ReconnectJob(ctx, jobId, nil); err != nil {\n    log.Printf(\"job unrecoverable, creating new job: %v\", err)\n    // fall back to launching a fresh job/block\n}","handlingStrategy":"fallback","validationCode":"// no local check can predict remote process death; handle after the fact\n","typeGuard":null,"tryCatchPattern":"if err := jobcontroller.ReconnectJob(ctx, jobId, nil); err != nil && strings.Contains(err.Error(), \"job manager has exited\") {\n    // unrecoverable: create a new job / notify user session is gone\n}","preventionTips":["Run long-lived remote jobs under tmux/systemd to survive daemon restarts","Treat JobManagerGone as terminal — do not retry","Persist a fallback action (new job) for jobs whose manager dies"],"tags":["go","remote-jobs","process-lifecycle"],"backgroundTag":"job-manager-gone","analyzedSha":"a4447c1563b2df285ab89e76c82f91e1a1a49c1e","analyzedAt":"2026-09-01T15:26:23.972Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}