hcengineering/platform · warning

high degraded sessions

Error message

high degraded sessions

What it means

This is a health-check warning (not a thrown exception) emitted by TSessionManager.checkHealth when the percentage of hung sessions exceeds hungSessionsWarnPercent but not hungSessionsFailPercent, causing the server to report status 'degraded'. A session counts as hung when its last request is older than hangSessionTimeoutSeconds, or when a large fraction of its in-flight requests exceed hangRequestTimeoutSeconds. The library emits it so operators can react before the server becomes fully unhealthy.

Source

Thrown at foundations/server/packages/server/src/sessionManager.ts:548

      }

      const hungRequestsPercent = totalRequests > 0 ? (100 * hungRequests) / totalRequests : 0
      if (hungRequestsPercent > this.hungRequestsFailPercent) {
        hungSessions += 1
      }
    }

    this.ctx.measure('sessions-hung', hungSessions, true)

    const hungSessionsPercent = totalSessions > 0 ? (100 * hungSessions) / totalSessions : 0

    if (hungSessionsPercent > this.hungSessionsFailPercent) {
      this.ctx.warn('high hung sessions', { hungSessionsPercent })
      return 'unhealthy'
    }

    if (hungSessionsPercent > this.hungSessionsWarnPercent) {
      this.ctx.warn('high degraded sessions', { hungSessionsPercent })
      return 'degraded'
    }

    return 'healthy'
  }

  tickCounter = 0

  @withContext('🧭 get-workspace')
  async getWorkspace (
    ctx: MeasureContext,
    workspaceUuid: WorkspaceUuid,
    workspaceInfo: LoginInfoWorkspace | undefined,
    token: Token,
    ws: ConnectionSocket
  ): Promise<{ workspace?: Workspace, resp?: GetWorkspaceResponse }> {
    if (workspaceInfo === undefined) {
      return { resp: { error: new Error('Workspace not found or not available'), terminate: true } }

View on GitHub (pinned to 63e28dc964)

Solutions

  1. Inspect which sessions are hung via getStatistics and check their lastRequest timestamps and in-flight request durations.
  2. Check downstream dependencies (database, storage, queue) for latency or lock contention slowing requests.
  3. Tune hangSessionWarnPercent / hangSessionTimeoutSeconds / hangRequestTimeoutSeconds configuration if the workload legitimately has long requests.
  4. Restart or drain the affected sessions/workspaces; if degradation persists, investigate for memory leaks or stuck event-loop tasks.

Example fix

// before: defaults too tight for a migration-heavy workload
sessionManager = new TSessionManager(ctx, ..., { hangSessionTimeoutSeconds: 30 })
// after
sessionManager = new TSessionManager(ctx, ..., { hangSessionTimeoutSeconds: 300, hangRequestTimeoutSeconds: 120 })
Defensive patterns

Strategy: validation

Validate before calling

const stats = await sessionManager.getStatistics()
const health = await sessionManager.checkHealth()
if (health !== 'healthy') {
  console.warn('server degraded before accepting load:', health)
}

Type guard

function isHealthy(status: string): boolean {
  return status === 'healthy'
}

Try / catch

try {
  const status = await sessionManager.checkHealth()
  if (status !== 'healthy') await alertOps(status)
} catch (err) {
  logger.error('health check failed', { err })
}

Prevention

When it happens

Trigger: checkHealth (invoked by health, getStatistics, status endpoints) computes hungSessionsPercent > this.hungSessionsWarnPercent while hungSessionsPercent <= this.hungSessionsFailPercent. Sessions are counted hung if Date.now() - session.lastRequest > hangSessionTimeoutSeconds * 1000, or if >hungRequestsFailPercent of a session's live requests have run longer than hangRequestTimeoutSeconds. System-account sessions are excluded.

Common situations: Long-running CPU-heavy operations (builds, migrations) blocking client requests; clients that went idle/disconnected without their sessions being cleaned; a backend dependency (DB, storage) slowing requests past hangRequestTimeoutSeconds; timeouts configured too aggressively for the workload.

Related errors

AI-assisted analysis of hcengineering/platform@63e28dc964 (2026-08-29). Data as JSON: /api/errors/868e45638fab9eff. Report an issue: GitHub.