{"record":{"id":"62f11320557e552b","repo":"overleaf/overleaf","slug":"internal-server-error-redis-pubsub-health-check-f","errorCode":null,"errorMessage":"Internal Server Error (redis/pubsub health check failed res.sendStatus(500))","messagePattern":"Internal Server Error \\(redis/pubsub health check failed res\\.sendStatus\\(500\\)\\)","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"services/real-time/app.js","lineNumber":169,"sourceCode":"    res.sendStatus(503) // Service unavailable\n  } else {\n    res.send('real-time is alive')\n  }\n})\n\napp.get('/debug/events', function (req, res) {\n  Settings.debugEvents = parseInt(req.query.count, 10) || 20\n  logger.info({ count: Settings.debugEvents }, 'starting debug mode')\n  res.send(`debug mode will log next ${Settings.debugEvents} events`)\n})\n\nconst rclient = redis.createClient(Settings.redis.realtime)\n\nfunction healthCheck(req, res) {\n  rclient.healthCheck(function (error) {\n    if (error) {\n      logger.err({ err: error }, 'failed redis health check')\n      res.sendStatus(500)\n    } else if (HealthCheckManager.isFailing()) {\n      const status = HealthCheckManager.status()\n      logger.err({ pubSubErrors: status }, 'failed pubsub health check')\n      res.sendStatus(500)\n    } else {\n      res.sendStatus(200)\n    }\n  })\n}\napp.get(\n  '/health_check',\n  (req, res, next) => {\n    if (Settings.shutDownComplete) {\n      return res.sendStatus(503)\n    }\n    next()\n  },\n  healthCheck","sourceCodeStart":151,"sourceCodeEnd":187,"githubUrl":"https://github.com/overleaf/overleaf/blob/28ad3b03b71cb4311decdcb55c36b33ec10d72db/services/real-time/app.js#L151-L187","documentation":"GET /health_check returns HTTP 500 when either the redis client healthCheck fails or HealthCheckManager reports accumulated pub/sub errors. The real-time service depends on redis pub/sub to fan out messages, so a broken connection makes the instance ineligible for traffic.","triggerScenarios":"rclient.healthCheck callback receives an error (redis unreachable/auth failure), or HealthCheckManager.isFailing() is true because pub/sub message processing is erroring.","commonSituations":"Redis restart or failover in the cluster; wrong Settings.redis.realtime host/port/password; network policy blocking the pod from redis; persistent pub/sub handler exceptions flipping HealthCheckManager into failing state.","solutions":["Check redis connectivity and credentials in Settings.redis.realtime; verify with redis-cli ping from the pod","Inspect logs for 'failed redis health check' or 'failed pubsub health check' for the underlying error","If HealthCheckManager.isFailing(), look at recent pub/sub errors and reconnect logic","Scale/replace the pod and confirm redis is sized correctly for the cluster"],"exampleFix":"// before\nrclient.healthCheck(function (error) {\n  if (error) { res.sendStatus(500) }\n})\n// after\n// surface the cause in the response body for faster triage\nrclient.healthCheck(function (error) {\n  if (error) {\n    res.status(500).json({ error: 'redis-unreachable', detail: error.message })\n  }\n})","handlingStrategy":"retry","validationCode":"const redisUp = await new Promise((ok, bad) => rclient.ping(e => (e ? bad(e) : ok(true))))","typeGuard":null,"tryCatchPattern":"try {\n  const res = await fetch(healthCheckUrl)\n  if (res.status === 500) {\n    logger.error('real-time redis/pubsub unhealthy; retrying')\n    return retryWithBackoff()\n  }\n} catch (err) {\n  logger.warn({ err }, 'health check request failed')\n}","preventionTips":["Monitor redis latency and connection counts from real-time pods","Alert on HealthCheckManager pub/sub error accumulation before it flips the endpoint","Pin and test redis credentials/connection settings per environment","Set redis client reconnect strategy so transient outages self-heal"],"tags":["redis","pubsub","health-check","http-500"],"backgroundTag":"redis-connection-failed","analyzedSha":"28ad3b03b71cb4311decdcb55c36b33ec10d72db","analyzedAt":"2026-09-03T02:10:22.807Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-10T07:17:11.731Z"}