{"record":{"id":"3d56e3b8bffb8ab2","repo":"wavetermdev/waveterm","slug":"failed-to-reconnect-to-job-manager-w","errorCode":null,"errorMessage":"failed to reconnect to job manager: %w","messagePattern":"failed to reconnect to job manager: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/jobcontroller/jobcontroller.go","lineNumber":1118,"sourceCode":"\n\treconnectData := wshrpc.CommandRemoteReconnectToJobManagerData{\n\t\tJobId:              jobId,\n\t\tJobAuthToken:       job.JobAuthToken,\n\t\tMainServerJwtToken: jobAccessToken,\n\t\tJobManagerPid:      job.JobManagerPid,\n\t\tJobManagerStartTs:  job.JobManagerStartTs,\n\t}\n\n\trpcOpts := &wshrpc.RpcOpts{\n\t\tRoute:   wshutil.MakeConnectionRouteId(job.Connection),\n\t\tTimeout: 5000,\n\t}\n\n\tlog.Printf(\"[job:%s] sending RemoteReconnectToJobManagerCommand to connection %s\", jobId, job.Connection)\n\trtnData, err := wshclient.RemoteReconnectToJobManagerCommand(bareRpc, reconnectData, rpcOpts)\n\tif err != nil {\n\t\tlog.Printf(\"[job:%s] RemoteReconnectToJobManagerCommand failed: %v\", jobId, err)\n\t\treturn fmt.Errorf(\"failed to reconnect to job manager: %w\", err)\n\t}\n\n\tif !rtnData.Success {\n\t\tlog.Printf(\"[job:%s] RemoteReconnectToJobManagerCommand returned error: %s\", jobId, rtnData.Error)\n\t\tif rtnData.JobManagerGone {\n\t\t\tvar updatedJob *waveobj.Job\n\t\t\tupdateErr := wstore.DBUpdateFn(ctx, jobId, func(job *waveobj.Job) {\n\t\t\t\tjob.JobManagerStatus = JobManagerStatus_Done\n\t\t\t\tjob.JobManagerDoneReason = JobDoneReason_Gone\n\t\t\t\tupdatedJob = job\n\t\t\t})\n\t\t\tif updateErr != nil {\n\t\t\t\tlog.Printf(\"[job:%s] error updating job manager running status: %v\", jobId, updateErr)\n\t\t\t} else {\n\t\t\t\tsendBlockJobStatusEventByJob(ctx, updatedJob)\n\t\t\t}\n\t\t\ttelemetry.GoRecordTEventWrap(&telemetrydata.TEvent{\n\t\t\t\tEvent: \"job:done\",","sourceCodeStart":1100,"sourceCodeEnd":1136,"githubUrl":"https://github.com/wavetermdev/waveterm/blob/a4447c1563b2df285ab89e76c82f91e1a1a49c1e/pkg/jobcontroller/jobcontroller.go#L1100-L1136","documentation":"This error is returned when the RPC call wshclient.RemoteReconnectToJobManagerCommand fails at the transport/RPC level while trying to reconnect a job manager over its connection (jobcontroller.go:1118). It wraps the underlying error from the RPC round-trip, which may be a route error, timeout (5000ms here), or the remote peer rejecting the call.","triggerScenarios":"RemoteReconnectToJobManagerCommand returns an error: the connection route (MakeConnectionRouteId(job.Connection)) has no responder, the RPC timed out after 5s, or the bare RPC client lost its link while the conn still reports connected.","commonSituations":"1) Connection dropped between the IsConnected check and the RPC call (race). 2) Remote wsh daemon died so no route exists for the conn. 3) Slow/broken network causing the 5-second RPC timeout.","solutions":["Retry ReconnectJob; the reconnectGroup deduplicates concurrent attempts.","Verify the connection is still alive and reconnect it if needed, then retry.","Check logs for the underlying wrapped error (e.g. 'no route', 'timeout') to distinguish transport vs timeout.","Increase robustness by ensuring the remote wsh daemon is running on the target host."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"connected, _ := conncontroller.IsConnected(connName)\n// only attempt RPC reconnect when conn is live","typeGuard":null,"tryCatchPattern":"if err := jobcontroller.ReconnectJob(ctx, jobId, nil); err != nil && strings.Contains(err.Error(), \"failed to reconnect to job manager\") {\n    // transient RPC failure: retry with backoff\n}","preventionTips":["Retry reconnects with exponential backoff","Keep the connection alive during reconnect flows","Watch for RPC timeout errors distinguishing slow network from dead routes"],"tags":["go","rpc","network"],"backgroundTag":"rpc-call-failed","analyzedSha":"a4447c1563b2df285ab89e76c82f91e1a1a49c1e","analyzedAt":"2026-09-01T15:26:23.972Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}