What causes "reconnect workspace in upgrade" in hcengineering/platform?

Wait for the ongoing upgrade to finish; the reconnected upgrade client should resume/observe upgrade progress.

reconnect workspace in upgrade

Error message

reconnect workspace in upgrade

What it means

Warning logged by getWorkspace when a connection with token.extra?.model === 'upgrade' reconnects to a workspace that is already flagged maintenance (workspace.maintenance === true). It means an upgrade client reconnected while the workspace is mid-upgrade; the connection is allowed but no session switch is performed, unlike the non-maintenance case.

Source

Thrown at foundations/server/packages/server/src/sessionManager.ts:654

    const branding =
      (workspaceInfo.branding !== undefined
        ? Object.values(this.brandingMap).find((b) => b.key === workspaceInfo?.branding)
        : null) ?? null

    if (workspace === undefined) {
      ctx.warn('open workspace', {
        account: token.account,
        workspace: workspaceUuid,
        ...token.extra
      })

      workspace = this.createWorkspace(ctx.parent ?? ctx, ctx, token, workspaceInfo.url, workspaceInfo.dataId, branding)
      await this.workspaceProducer.send(ctx, workspaceUuid, [workspaceEvents.open()])
    }

    if (token.extra?.model === 'upgrade') {
      if (workspace.maintenance) {
        ctx.warn('reconnect workspace in upgrade', {
          account: token.account,
          workspace: workspaceUuid,
          wsUrl: workspaceInfo.url
        })
      } else {
        ctx.warn('reconnect workspace in upgrade switch', {
          email: token.account,
          workspace: workspaceUuid,
          wsUrl: workspaceInfo.url
        })

        // We need to wait in case previous upgrade connection is already closing.
        await this.switchToUpgradeSession(token, ctx.parent ?? ctx, workspace, ws)
      }
    } else {
      if (workspace.maintenance || this.maintenanceWorkspaces.has(workspace.wsId.uuid)) {
        ctx.warn('connect during upgrade', {
          account: token.account,

View on GitHub (pinned to 63e28dc964)

Solutions

Wait for the ongoing upgrade to finish; the reconnected upgrade client should resume/observe upgrade progress.
Verify the upgrade process (switchToUpgradeSession / upgrade pipeline) is still running and healthy; restart it if it died mid-upgrade leaving maintenance stuck.
If maintenance is stuck, use forceClose to fully tear the workspace down and let a fresh upgrade run.

Defensive patterns

Strategy: retry

Validate before calling

const ws = sessionManager.workspaces.get(wsUuid)
if (ws?.maintenance === true) {
  await waitForUpgradeCompletion(wsUuid) // poll upgrade progress before reconnecting upgrade client
}

Type guard

function isUpgrading(ws: Workspace | undefined): boolean {
  return ws !== undefined && ws.maintenance
}

Try / catch

try {
  await sessionManager.getWorkspace(ctx, wsUuid, info, upgradeToken, socket)
} catch (err) {
  if (isUpgradeInProgress(err)) {
    await delay(retryInterval)
    return retryUpgradeConnection(wsUuid)
  }
  throw err
}

Prevention

Ensure exactly one upgrade client per workspace; serialize upgrade attempts with a lock.
Make upgrade clients idempotent so a mid-upgrade reconnect resumes rather than restarts.
Alert if maintenance stays true longer than the expected upgrade duration (stuck upgrade).
Monitor upgrade client network stability; reconnects mid-upgrade are the main trigger.

When it happens

Trigger: getWorkspace with token.extra.model === 'upgrade' and the resolved workspace object already has maintenance=true — e.g., an upgrade client dropped mid-upgrade and reconnected, or forceMaintenance/forceClose set the maintenance flag and the upgrade connection re-attaches.

Common situations: Network drop during a model upgrade causing the upgrade client to reconnect; duplicate upgrade connections racing; a previous upgrade attempt ended abnormally leaving maintenance set.

Related errors

AI-assisted analysis of hcengineering/platform@63e28dc964 (2026-08-29). Data as JSON: /api/errors/d0313df179948e49. Report an issue: GitHub.