{"record":{"id":"358f786ddc0f8816","repo":"vitessio/vitess","slug":"healthcheck-timed-out-latest-v","errorCode":null,"errorMessage":"healthcheck timed out (latest %v)","messagePattern":"healthcheck timed out \\(latest (.+?)\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/discovery/tablet_health_check.go","lineNumber":311,"sourceCode":"\t\t\t// This means that another tablet has taken over the host:port that we were connected to.\n\t\t\t// So let's remove the tablet's data from the healthcheck, and if it is still a part of the\n\t\t\t// cluster, the new tablet record will be fetched from the topology server and re-added to\n\t\t\t// the healthcheck cache again via the topology watcher.\n\t\t\t// WARNING: Under no other circumstances should we be deleting the tablet here.\n\t\t\tif strings.Contains(err.Error(), \"health stats mismatch\") {\n\t\t\t\tthc.logger.Warningf(\"deleting tablet %v from healthcheck due to health stats mismatch\", thc.Tablet)\n\t\t\t\thc.deleteTablet(thc.Tablet)\n\t\t\t\treturn\n\t\t\t}\n\t\t\t// trivialUpdate = false because this is an error\n\t\t\t// up = false because we did not get a healthy response\n\t\t\thc.updateHealth(thc, thc.Target, false, false)\n\t\t}\n\t\t// If there was a timeout send an error. We do this after stream has returned.\n\t\t// This will ensure that this update prevails over any previous message that\n\t\t// stream could have sent.\n\t\tif timedout.Load() {\n\t\t\tthc.LastError = fmt.Errorf(\"healthcheck timed out (latest %v)\", thc.lastResponseTimestamp)\n\t\t\tthc.setServingState(false, thc.LastError.Error())\n\t\t\thcErrorCounters.Add([]string{thc.Target.Keyspace, thc.Target.Shard, topoproto.TabletTypeLString(thc.Target.TabletType)}, 1)\n\t\t\t// trivialUpdate = false because this is an error\n\t\t\t// up = false because we did not get a healthy response within the timeout\n\t\t\thc.updateHealth(thc, thc.Target, false, false)\n\t\t}\n\n\t\t// Streaming RPC failed e.g. because vttablet was restarted or took too long.\n\t\t// Sleep until the next retry is up or the context is done/canceled.\n\t\tselect {\n\t\tcase <-thc.ctx.Done():\n\t\t\treturn\n\t\tcase <-time.After(retryDelay):\n\t\t\t// Exponentially back-off to prevent tight-loop.\n\t\t\tretryDelay = nextHealthCheckRetryDelay(retryDelay)\n\t\t}\n\t}\n}","sourceCodeStart":293,"sourceCodeEnd":329,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/discovery/tablet_health_check.go#L293-L329","documentation":"checkConn streams health from a vttablet; if no health response arrives within the configured timeout, the healthcheck marks the tablet as not serving and records this error (with the last response timestamp) as the tablet's LastError. It is emitted after the stream returns so the timeout verdict overrides any earlier status.","triggerScenarios":"A vttablet fails to answer StreamHealth within healthcheck timeouts (default ~ healthcheck interval/timeout flags) — slow tablet, overloaded MySQL, network partition, or a hung gRPC stream; observed via AddTablet's initial check and subsequent checkConn loops.","commonSituations":"Tablet overloaded or blocked on MySQL queries so StreamHealth can't respond; firewall silently dropping the gRPC connection; tablet host paused (VM freeze) or in GC/network stall; too-aggressive timeout flags in large clusters.","solutions":["Check network connectivity and latency between the healthcheck process (VTGate/vtctld) and the tablet's gRPC port.","Inspect the tablet for overload: MySQL slow queries, lock contention, CPU saturation.","Increase --healthcheck-timeout / related flags if legitimate but slow tablets are being marked unhealthy.","Confirm vttablet process is alive and its StreamHealth RPCs are being served (tablet logs, metrics)."],"exampleFix":"// before\nvtgate --healthcheck-timeout 1s   # too aggressive\n// after\nvtgate --healthcheck-timeout 5s   # tolerate slow but healthy tablets","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"healthcheck timed out\") {\n\t// tablet did not answer within timeout; probe before routing traffic\n\tif probeTablet(alias, probeTimeout) {\n\t\treturn retryRouting(alias)\n\t}\n\treturn routeAround(alias)\n}","preventionTips":["Size healthcheck timeouts for worst-case tablet latency, not average","Monitor gRPC connectivity and latency between gateway and tablets","Watch for tablet overload (MySQL slow queries) that blocks StreamHealth"],"tags":["healthcheck","timeout","network"],"backgroundTag":"health-check-timeout","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}