wavetermdev/waveterm · critical

error authenticating with upstream

Error message

error authenticating with upstream: %v

What it means

This wraps the failure of wshclient.AuthenticateCommand, which sends the JWT to the upstream router's control RPC (ControlRootRoute) to authenticate the domain-socket connection. Failure means the upstream rejected the token (bad signature, expired, wrong audience) or the RPC itself failed, so the connserver refuses to continue without verified credentials.

Solutions

1. Restart the wsh connserver so it gets a fresh JWT from the Wave terminal environment.
2. Check system clock skew (NTP) between client and server.
3. Ensure wsh and Wave server are the same version (signing/verification changes between releases).
4. Look at the wrapped %v: 'token expired' → new token; 'signature invalid' → version/key mismatch; 'connection closed' → upstream socket died.
5. Remove stale socket files and restart both Wave server and wsh connserver.

// before authenticating, sanity-check token expiry (unverified)
parts := strings.Split(jwtToken, ".")
if len(parts) == 3 {
	if raw, err := base64.RawURLEncoding.DecodeString(parts[1]); err == nil {
		var c struct{ Exp int64 `json:"exp"` }
		if json.Unmarshal(raw, &c) == nil && c.Exp != 0 && time.Now().Unix() > c.Exp {
			return errors.New("jwt already expired; get a fresh token")
		}
	}
}

_, err = wshclient.AuthenticateCommand(controlRpc, jwtToken, &wshrpc.RpcOpts{Route: wshutil.ControlRootRoute})
if err != nil {
	return fmt.Errorf("error authenticating with upstream: %w", err)
}

When it happens

Trigger: wshclient.AuthenticateCommand(controlRpc, jwtToken, {Route: ControlRootRoute}) returns an error — the upstream router denied the JWT or the RPC timed out/disconnected.

Common situations: Clock skew making the JWT appear expired; JWT generated by a different Wave install with different signing keys; stale token reused after the server restarted with new secrets; upstream dropped the connection mid-RPC.

Understand the failure class

• Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

• error authenticating rpc connection
• error getting jwt public key
• failed to authenticate to server
• failed to validate token
• JobId mismatch

AI-assisted analysis of wavetermdev/waveterm@a4447c1563 (2026-09-01). Data as JSON: /api/errors/b5503b774898fa28. Report an issue: GitHub.

Appendix: source

Thrown at cmd/wsh/cmd/wshcmd-connserver.go:343

			panichandler.PanicHandler("serverRunRouterDomainSocket:ReadLoop", recover())
		}()
		defer func() {
			log.Printf("upstream domain socket closed, shutting down")
			wshutil.DoShutdown("", 0, true)
		}()
		wshutil.AdaptStreamToMsgCh(conn, upstreamProxy.FromRemoteCh, nil)
	}()

	// register the domain socket connection as upstream
	router.RegisterUpstream(upstreamProxy)

	// use the router's control RPC to authenticate with upstream
	controlRpc := router.GetControlRpc()

	// authenticate with the upstream router using the JWT
	_, err = wshclient.AuthenticateCommand(controlRpc, jwtToken, &wshrpc.RpcOpts{Route: wshutil.ControlRootRoute})
	if err != nil {
		return fmt.Errorf("error authenticating with upstream: %v", err)
	}
	log.Printf("authenticated with upstream router")

	// fetch and set JWT public key
	log.Printf("trying to get JWT public key")
	jwtPublicKeyB64, err := wshclient.GetJwtPublicKeyCommand(controlRpc, nil)
	if err != nil {
		return fmt.Errorf("error getting jwt public key: %v", err)
	}
	jwtPublicKeyBytes, err := base64.StdEncoding.DecodeString(jwtPublicKeyB64)
	if err != nil {
		return fmt.Errorf("error decoding jwt public key: %v", err)
	}
	err = wavejwt.SetPublicKey(jwtPublicKeyBytes)
	if err != nil {
		return fmt.Errorf("error setting jwt public key: %v", err)
	}
	log.Printf("got JWT public key")

View on GitHub (pinned to a4447c1563)