hcengineering/platform · warning

high degraded sessions

Error message

high degraded sessions

What it means

This is a health-check warning (not a thrown exception) emitted by TSessionManager.checkHealth when the percentage of hung sessions exceeds hungSessionsWarnPercent but not hungSessionsFailPercent, causing the server to report status 'degraded'. A session counts as hung when its last request is older than hangSessionTimeoutSeconds, or when a large fraction of its in-flight requests exceed hangRequestTimeoutSeconds. The library emits it so operators can react before the server becomes fully unhealthy.

Solutions

  1. Inspect which sessions are hung via getStatistics and check their lastRequest timestamps and in-flight request durations.
  2. Check downstream dependencies (database, storage, queue) for latency or lock contention slowing requests.
  3. Tune hangSessionWarnPercent / hangSessionTimeoutSeconds / hangRequestTimeoutSeconds configuration if the workload legitimately has long requests.
  4. Restart or drain the affected sessions/workspaces; if degradation persists, investigate for memory leaks or stuck event-loop tasks.

Example fix

// before: defaults too tight for a migration-heavy workload
sessionManager = new TSessionManager(ctx, ..., { hangSessionTimeoutSeconds: 30 })
// after
sessionManager = new TSessionManager(ctx, ..., { hangSessionTimeoutSeconds: 300, hangRequestTimeoutSeconds: 120 })
Defensive patterns

Strategy: validation

Validate before calling

const stats = await sessionManager.getStatistics()
const health = await sessionManager.checkHealth()
if (health !== 'healthy') {
  console.warn('server degraded before accepting load:', health)
}

Type guard

function isHealthy(status: string): boolean {
  return status === 'healthy'
}

Try / catch

try {
  const status = await sessionManager.checkHealth()
  if (status !== 'healthy') await alertOps(status)
} catch (err) {
  logger.error('health check failed', { err })
}

Prevention

When it happens

Trigger: checkHealth (invoked by health, getStatistics, status endpoints) computes hungSessionsPercent > this.hungSessionsWarnPercent while hungSessionsPercent <= this.hungSessionsFailPercent. Sessions are counted hung if Date.now() - session.lastRequest > hangSessionTimeoutSeconds * 1000, or if >hungRequestsFailPercent of a session's live requests have run longer than hangRequestTimeoutSeconds. System-account sessions are excluded.

Common situations: Long-running CPU-heavy operations (builds, migrations) blocking client requests; clients that went idle/disconnected without their sessions being cleaned; a backend dependency (DB, storage) slowing requests past hangRequestTimeoutSeconds; timeouts configured too aggressively for the workload.

Related errors

AI-assisted analysis of hcengineering/platform@63e28dc964 (2026-08-29). Data as JSON: /api/errors/868e45638fab9eff. Report an issue: GitHub.

Appendix: source

Thrown at foundations/server/packages/server/src/sessionManager.ts:548

      }

      const hungRequestsPercent = totalRequests > 0 ? (100 * hungRequests) / totalRequests : 0
      if (hungRequestsPercent > this.hungRequestsFailPercent) {
        hungSessions += 1
      }
    }

    this.ctx.measure('sessions-hung', hungSessions, true)

    const hungSessionsPercent = totalSessions > 0 ? (100 * hungSessions) / totalSessions : 0

    if (hungSessionsPercent > this.hungSessionsFailPercent) {
      this.ctx.warn('high hung sessions', { hungSessionsPercent })
      return 'unhealthy'
    }

    if (hungSessionsPercent > this.hungSessionsWarnPercent) {
      this.ctx.warn('high degraded sessions', { hungSessionsPercent })
      return 'degraded'
    }

    return 'healthy'
  }

  tickCounter = 0

  @withContext('🧭 get-workspace')
  async getWorkspace (
    ctx: MeasureContext,
    workspaceUuid: WorkspaceUuid,
    workspaceInfo: LoginInfoWorkspace | undefined,
    token: Token,
    ws: ConnectionSocket
  ): Promise<{ workspace?: Workspace, resp?: GetWorkspaceResponse }> {
    if (workspaceInfo === undefined) {
      return { resp: { error: new Error('Workspace not found or not available'), terminate: true } }

View on GitHub (pinned to 63e28dc964)