{"record":{"id":"575388e56aec73a3","repo":"wavetermdev/waveterm","slug":"failed-to-reconnect-to-job-manager-s","errorCode":null,"errorMessage":"failed to reconnect to job manager: %s","messagePattern":"failed to reconnect to job manager: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/jobcontroller/jobcontroller.go","lineNumber":1145,"sourceCode":"\t\t\t\tjob.JobManagerDoneReason = JobDoneReason_Gone\n\t\t\t\tupdatedJob = job\n\t\t\t})\n\t\t\tif updateErr != nil {\n\t\t\t\tlog.Printf(\"[job:%s] error updating job manager running status: %v\", jobId, updateErr)\n\t\t\t} else {\n\t\t\t\tsendBlockJobStatusEventByJob(ctx, updatedJob)\n\t\t\t}\n\t\t\ttelemetry.GoRecordTEventWrap(&telemetrydata.TEvent{\n\t\t\t\tEvent: \"job:done\",\n\t\t\t\tProps: telemetrydata.TEventProps{\n\t\t\t\t\tJobDoneReason: JobDoneReason_Gone,\n\t\t\t\t\tJobKind:       job.JobKind,\n\t\t\t\t},\n\t\t\t})\n\t\t\twriteJobTerminationMessage(ctx, jobId, updatedJob, \"[session gone]\")\n\t\t\treturn fmt.Errorf(\"job manager has exited: %s\", rtnData.Error)\n\t\t}\n\t\treturn fmt.Errorf(\"failed to reconnect to job manager: %s\", rtnData.Error)\n\t}\n\n\tlog.Printf(\"[job:%s] RemoteReconnectToJobManagerCommand succeeded, waiting for route\", jobId)\n\n\trouteId := wshutil.MakeJobRouteId(jobId)\n\twaitCtx, cancelFn := context.WithTimeout(ctx, 2*time.Second)\n\tdefer cancelFn()\n\terr = wshutil.DefaultRouter.WaitForRegister(waitCtx, routeId)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"route did not establish after successful reconnection: %w\", err)\n\t}\n\tSetJobConnStatus(jobId, JobConnStatus_Connected)\n\tsendBlockJobStatusEventByJob(ctx, job)\n\n\ttelemetry.GoRecordTEventWrap(&telemetrydata.TEvent{\n\t\tEvent: \"job:reconnect\",\n\t\tProps: telemetrydata.TEventProps{\n\t\t\tJobKind: job.JobKind,","sourceCodeStart":1127,"sourceCodeEnd":1163,"githubUrl":"https://github.com/wavetermdev/waveterm/blob/a4447c1563b2df285ab89e76c82f91e1a1a49c1e/pkg/jobcontroller/jobcontroller.go#L1127-L1163","documentation":"This error is returned when the RemoteReconnectToJobManagerCommand RPC completed but the remote side answered Success=false without JobManagerGone (jobcontroller.go:1145). The remote daemon attempted to reattach to the job manager and failed for a reason other than the process being gone (e.g. auth token mismatch, pid reuse check failed). The remote's error string is included verbatim.","triggerScenarios":"rtnData.Success is false and rtnData.JobManagerGone is false — the remote explicitly rejected the reconnect with its own error message in rtnData.Error.","commonSituations":"1) Stale job.JobAuthToken / pid metadata that no longer matches the running manager. 2) The remote daemon restarted and partially lost state about the job manager. 3) Version mismatch between client and remote wsh during the reconnect handshake.","solutions":["Read rtnData.Error (embedded in the message) for the remote's specific failure reason.","Retry the reconnect; transient remote-state issues may resolve on a second attempt.","If persistent, terminate the job and start a new one since reattachment keeps failing.","Ensure client and remote wsh versions are compatible."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// ensure client and remote wsh versions match before reconnect","typeGuard":null,"tryCatchPattern":"if err := jobcontroller.ReconnectJob(ctx, jobId, nil); err != nil && strings.Contains(err.Error(), \"failed to reconnect to job manager\") && !strings.Contains(err.Error(), \"has exited\") {\n    // remote rejected: inspect embedded reason, optionally retry once\n}","preventionTips":["Keep wsh versions in sync between client and remote","Retry once on transient remote-state errors, then give up","Log the embedded remote error string for diagnosis"],"tags":["go","rpc","remote-jobs"],"backgroundTag":"job-reconnect-rejected","analyzedSha":"a4447c1563b2df285ab89e76c82f91e1a1a49c1e","analyzedAt":"2026-09-01T15:26:23.972Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}