vitessio/vitess · error

500 internal server error: vttablet is not serving

Error message

500 internal server error: vttablet is not serving

What it means

This is the HTTP 500 body returned by the vttablet /healthz endpoint when the tablet's state manager says it should be serving (wantState is StateServing or StateNotConnected) but the query service is not actually in the serving state. It signals that the tablet's health check is failing because the underlying query service has not transitioned into serving, typically during startup, shutdown, or after losing connection to MySQL. Load balancers and vtgates use this endpoint to stop routing traffic to the tablet.

Source

Thrown at go/vt/vttablet/tabletserver/tabletserver.go:2083

// Close shuts down any remaining go routines
func (tsv *TabletServer) Close(ctx context.Context) error {
	tsv.sm.closeAll()
	tsv.stats.Stop()
	return nil
}

var okMessage = []byte("ok\n")

// Health check
// Returns ok if we are in the desired serving state
func (tsv *TabletServer) registerHealthzHealthHandler() {
	tsv.exporter.HandleFunc("/healthz", tsv.healthzHandler)
}

func (tsv *TabletServer) healthzHandler(w http.ResponseWriter, r *http.Request) {
	if (tsv.sm.wantState == StateServing || tsv.sm.wantState == StateNotConnected) && !tsv.sm.IsServing() {
		http.Error(w, "500 internal server error: vttablet is not serving", http.StatusInternalServerError)
		return
	}
	w.Header().Set("Content-Length", strconv.Itoa(len(okMessage)))
	w.WriteHeader(http.StatusOK)
	w.Write(okMessage)
}

// Query service health check
// Returns ok if a query can go all the way to database and back
func (tsv *TabletServer) registerDebugHealthHandler() {
	tsv.exporter.HandleFunc("/debug/health", func(w http.ResponseWriter, r *http.Request) {
		if err := acl.CheckAccessHTTP(r, acl.MONITORING); err != nil {
			acl.SendError(w, err)
			return
		}
		w.Header().Set("Content-Type", "text/plain")
		if err := tsv.IsHealthy(); err != nil {
			http.Error(w, fmt.Sprintf("not ok: %v", err), http.StatusInternalServerError)

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check the vttablet logs and its /debug/vars (or tablet status page) to see why the query service is not serving — commonly MySQL connectivity — and fix the underlying cause (start/repair mysqld, fix connection params).
2. Wait for the tablet to finish transitioning to serving (re-run /healthz until it returns 200) or remove the tablet from the load balancer pool until serving.
3. If the tablet is stuck non-serving, restart vttablet or issue the appropriate SetServing/reparent workflow to transition it back to SERVING.
4. Verify the tablet's tablet type/state in vtctldclient (gettablet) and make sure a graceful shutdown or reparent is not in progress.

Example fix

// before: prober treats one failing probe as fatal
if !strings.Contains(resp.Body, "ok") {
    return fmt.Errorf("tablet unhealthy")
}
// after: tolerate transient non-serving during startup/restart
if !strings.Contains(resp.Body, "ok") {
    if errors.Is(err, ErrNotServing) || withinGracePeriod {
        return ErrRetryLater // allow retries during startup/failover
    }
    return fmt.Errorf("tablet unhealthy")
}

// Check tablet state before routing traffic
resp, _ := http.Get("http://tablet:15001/healthz")
if resp == nil || resp.StatusCode != http.StatusOK {
    // tablet not serving; exclude from pool or retry after backoff
    return ErrTabletNotReady
}

// Go: retry transient not-serving with bounded backoff
var lastErr error
for i := 0; i < 5; i++ {
    err := exec.Query(ctx, q)
    if err == nil {
        return nil
    }
    lastErr = err
    if strings.Contains(err.Error(), "vttablet is not serving") {
        time.Sleep(backoff(i))
        continue
    }
    break
}
return lastErr

When it happens

Trigger: An HTTP GET to the vttablet /healthz endpoint while tsv.sm.wantState is StateServing or StateNotConnected and tsv.sm.IsServing() returns false — e.g. the tablet has been told to serve but SetServingType has not completed, or the tabletserver was transitioned to non-serving (e.g. MySQL down, throttle, shutdown) without wantState yet being updated.

Common situations: vttablet is still starting up or restarting while a load balancer probes /healthz; MySQL/backup is down so the tabletserver stopped serving; a reparent or tablet restart races with health probes; monitoring scripts polling /healthz during a vttabletd graceful restart.

Understand the failure class

• HTTP status errors: handling 4xx and 5xx responses — how to handle 4xx and 5xx responses properly.

Related errors

• not ok: %v
• BeforeSchema differs
• AfterSchema differs
• the <tablet alias> argument is required for the RunHealthChe
• must be non-negative

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/8fe7830cc1659756. Report an issue: GitHub.