wavetermdev/waveterm · error

failed to reconnect to job

Error message

failed to reconnect to job: %w

What it means

DurableShellController.Start wraps any error from jobcontroller.ReconnectJob, which attempts to reattach the block to a previously created durable remote shell job. It means the existing job could not be resumed — the job may have died, its domain socket is gone, or the connection to the remote is down. The block's durable shell cannot be restored to its prior state.

Solutions

1. Check/refresh the SSH connection to the remote host (test with a plain ssh command).
2. Restart the block or call Start with force to create a fresh job instead of reconnecting.
3. Inspect the wrapped inner error (%w) for the root cause (socket not found vs auth vs network).
4. If the remote was rebooted, accept the session is lost and start a new shell.

Example fix

// before
err := controller.Start(ctx, blockMeta, rtOpts, false) // tries reconnect, fails
// after
if err := controller.Start(ctx, blockMeta, rtOpts, false); err != nil {
    // force a brand-new job instead of reconnecting to a dead one
    err = controller.Start(ctx, blockMeta, rtOpts, true)
}

// best-effort pre-check: is the remote reachable?
if conn := conncontroller.MaybeGetConn(opts); conn == nil || conn.DeriveConnStatus().Status != conncontroller.Status_Connected {
    return fmt.Errorf("remote not connected; reconnect before Start")
}

if err := dsc.Start(ctx, meta, rtOpts, false); err != nil {
    var root = errors.Unwrap(err) // inspect ReconnectJob cause
    log.Printf("reconnect failed (%v); forcing new job", root)
    err = dsc.Start(ctx, meta, rtOpts, true) // fallback to fresh job
}

When it happens

Trigger: Calling Start on a block whose meta indicates a durable remote shell with an existing JobId, when ReconnectJob fails — e.g. the SSH connection is down, the remote wsh server/socket no longer exists, or the job was already terminated on the remote side.

Common situations: Laptop wakes from sleep and the SSH connection dropped; the remote machine rebooted and the shell job is gone; stale JobId persisted in block state after a remote crash; connserver on the remote is not running.

Related errors

• connection not found
• error getting job manager status
• failed to get stderr pipe
• failed to start durable shell
• failed to start new job

AI-assisted analysis of wavetermdev/waveterm@a4447c1563 (2026-09-01). Data as JSON: /api/errors/5d1315a3ec4e0191. Report an issue: GitHub.

Appendix: source

Thrown at pkg/blockcontroller/durableshellcontroller.go:186

		fsErr := filestore.WFS.MakeFile(ctx, dsc.BlockId, wavebase.BlockFile_Term, nil, wshrpc.FileOpts{MaxSize: DefaultTermMaxFileSize, Circular: true})
		if fsErr != nil && fsErr != fs.ErrExist {
			return fmt.Errorf("error creating block term file: %w", fsErr)
		}
		newJobId, err := dsc.startNewJob(ctx, blockMeta, dsc.ConnName, rtOpts)
		if err != nil {
			return fmt.Errorf("failed to start new job: %w", err)
		}
		jobId = newJobId
	}

	dsc.WithLock(func() {
		dsc.JobId = jobId
		dsc.sendUpdate_withlock()
	})

	err = jobcontroller.ReconnectJob(ctx, jobId, rtOpts)
	if err != nil {
		return fmt.Errorf("failed to reconnect to job: %w", err)
	}

	return nil
}

func (dsc *DurableShellController) Stop(graceful bool, newStatus string, destroy bool) {
	if !destroy {
		return
	}
	jobId := dsc.getJobId()
	if jobId == "" {
		return
	}
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()
	jobcontroller.TerminateAndDetachJob(ctx, jobId)
}

View on GitHub (pinned to a4447c1563)