hcengineering/platform · error

high hung sessions

Error message

high hung sessions

What it means

Health check computes the percentage of hung sessions versus total sessions; if it exceeds hungSessionsFailPercent the server logs this warning and reports itself 'unhealthy' (a warn-level threshold reports 'degraded'). This is the instance-level health signal consumed by load balancers/orchestrators to stop routing traffic to a sick node.

Source

Thrown at foundations/server/packages/server/src/sessionManager.ts:543

        const sec = Math.round((now - r.start) / 1000)
        if (sec > hangRequestTimeoutSeconds) {
          hungRequests += 1
        }
        totalRequests += 1
      }

      const hungRequestsPercent = totalRequests > 0 ? (100 * hungRequests) / totalRequests : 0
      if (hungRequestsPercent > this.hungRequestsFailPercent) {
        hungSessions += 1
      }
    }

    this.ctx.measure('sessions-hung', hungSessions, true)

    const hungSessionsPercent = totalSessions > 0 ? (100 * hungSessions) / totalSessions : 0

    if (hungSessionsPercent > this.hungSessionsFailPercent) {
      this.ctx.warn('high hung sessions', { hungSessionsPercent })
      return 'unhealthy'
    }

    if (hungSessionsPercent > this.hungSessionsWarnPercent) {
      this.ctx.warn('high degraded sessions', { hungSessionsPercent })
      return 'degraded'
    }

    return 'healthy'
  }

  tickCounter = 0

  @withContext('🧭 get-workspace')
  async getWorkspace (
    ctx: MeasureContext,
    workspaceUuid: WorkspaceUuid,
    workspaceInfo: LoginInfoWorkspace | undefined,

View on GitHub (pinned to 63e28dc964)

Solutions

  1. Investigate cause of mass hangs: check 'session hang, closing...' and 'request hang found' logs and downstream service health
  2. Restart/replace the unhealthy instance so the orchestrator reschedules traffic
  3. Verify downstream dependencies (DB, storage, Kafka) are reachable from the server
  4. Tune hungSessionsFailPercent/hungSessionsWarnPercent thresholds to match realistic client behavior

Example fix

// before
health probe: strict 90% fail threshold, no LB drain
// after
WORKSPACE_HUNG_SESSIONS_FAIL_PERCENT=50
# and configure LB to respect /health status before rerouting
Defensive patterns

Strategy: validation

Validate before calling

const stats = await fetch(`${serverUrl}/status`).then(r => r.json())
if (stats.hungSessionsPercent > 50) {
  console.error('node unhealthy, rerouting traffic')
}

Type guard

function isUnhealthy(h: { hungSessionsPercent: number; failPercent: number }): boolean {
  return h.hungSessionsPercent > h.failPercent
}

Try / catch

try {
  const health = await getHealth()
  if (health === 'unhealthy') {
    await drainAndRestart()
  }
} catch (err) {
  logger.error('health check failed', { err })
}

Prevention

When it happens

Trigger: checkHealth (invoked by health, getStatistics, status endpoints) measures hungSessions and finds hungSessionsPercent > hungSessionsFailPercent — many clients stopped responding or many requests are stuck.

Common situations: Network outage on the client side leaving many dead sockets; server-side dependency outage causing mass request hangs; load balancer not draining connections; too-strict hungSessionsFailPercent for workloads with legitimately slow clients.

Related errors

AI-assisted analysis of hcengineering/platform@63e28dc964 (2026-08-29). Data as JSON: /api/errors/ab73fd875536c0af. Report an issue: GitHub.