vitessio/vitess · error

healthcheck timed out (latest %v)

Error message

healthcheck timed out (latest %v)

What it means

checkConn streams health from a vttablet; if no health response arrives within the configured timeout, the healthcheck marks the tablet as not serving and records this error (with the last response timestamp) as the tablet's LastError. It is emitted after the stream returns so the timeout verdict overrides any earlier status.

Source

Thrown at go/vt/discovery/tablet_health_check.go:311

			// This means that another tablet has taken over the host:port that we were connected to.
			// So let's remove the tablet's data from the healthcheck, and if it is still a part of the
			// cluster, the new tablet record will be fetched from the topology server and re-added to
			// the healthcheck cache again via the topology watcher.
			// WARNING: Under no other circumstances should we be deleting the tablet here.
			if strings.Contains(err.Error(), "health stats mismatch") {
				thc.logger.Warningf("deleting tablet %v from healthcheck due to health stats mismatch", thc.Tablet)
				hc.deleteTablet(thc.Tablet)
				return
			}
			// trivialUpdate = false because this is an error
			// up = false because we did not get a healthy response
			hc.updateHealth(thc, thc.Target, false, false)
		}
		// If there was a timeout send an error. We do this after stream has returned.
		// This will ensure that this update prevails over any previous message that
		// stream could have sent.
		if timedout.Load() {
			thc.LastError = fmt.Errorf("healthcheck timed out (latest %v)", thc.lastResponseTimestamp)
			thc.setServingState(false, thc.LastError.Error())
			hcErrorCounters.Add([]string{thc.Target.Keyspace, thc.Target.Shard, topoproto.TabletTypeLString(thc.Target.TabletType)}, 1)
			// trivialUpdate = false because this is an error
			// up = false because we did not get a healthy response within the timeout
			hc.updateHealth(thc, thc.Target, false, false)
		}

		// Streaming RPC failed e.g. because vttablet was restarted or took too long.
		// Sleep until the next retry is up or the context is done/canceled.
		select {
		case <-thc.ctx.Done():
			return
		case <-time.After(retryDelay):
			// Exponentially back-off to prevent tight-loop.
			retryDelay = nextHealthCheckRetryDelay(retryDelay)
		}
	}
}

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check network connectivity and latency between the healthcheck process (VTGate/vtctld) and the tablet's gRPC port.
2. Inspect the tablet for overload: MySQL slow queries, lock contention, CPU saturation.
3. Increase --healthcheck-timeout / related flags if legitimate but slow tablets are being marked unhealthy.
4. Confirm vttablet process is alive and its StreamHealth RPCs are being served (tablet logs, metrics).

Example fix

// before
vtgate --healthcheck-timeout 1s   # too aggressive
// after
vtgate --healthcheck-timeout 5s   # tolerate slow but healthy tablets

if err != nil && strings.Contains(err.Error(), "healthcheck timed out") {
	// tablet did not answer within timeout; probe before routing traffic
	if probeTablet(alias, probeTimeout) {
		return retryRouting(alias)
	}
	return routeAround(alias)
}

When it happens

Trigger: A vttablet fails to answer StreamHealth within healthcheck timeouts (default ~ healthcheck interval/timeout flags) — slow tablet, overloaded MySQL, network partition, or a hung gRPC stream; observed via AddTablet's initial check and subsequent checkConn loops.

Common situations: Tablet overloaded or blocked on MySQL queries so StreamHealth can't respond; firewall silently dropping the gRPC connection; tablet host paused (VM freeze) or in GC/network stall; too-aggressive timeout flags in large clusters.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• gRPC connection wait time exceeded
• failed to discover %ss (cluster %s): %w
• failed to run explicit healthcheck on tablet: %v err: %v
• invalid choice for enum
• deadline exceeded waiting for mysqld socket file to appear:

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/358f786ddc0f8816. Report an issue: GitHub.