{"record":{"id":"8fe7830cc1659756","repo":"vitessio/vitess","slug":"500-internal-server-error-vttablet-is-not-serving","errorCode":null,"errorMessage":"500 internal server error: vttablet is not serving","messagePattern":"500 internal server error: vttablet is not serving","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"go/vt/vttablet/tabletserver/tabletserver.go","lineNumber":2083,"sourceCode":"\n// Close shuts down any remaining go routines\nfunc (tsv *TabletServer) Close(ctx context.Context) error {\n\ttsv.sm.closeAll()\n\ttsv.stats.Stop()\n\treturn nil\n}\n\nvar okMessage = []byte(\"ok\\n\")\n\n// Health check\n// Returns ok if we are in the desired serving state\nfunc (tsv *TabletServer) registerHealthzHealthHandler() {\n\ttsv.exporter.HandleFunc(\"/healthz\", tsv.healthzHandler)\n}\n\nfunc (tsv *TabletServer) healthzHandler(w http.ResponseWriter, r *http.Request) {\n\tif (tsv.sm.wantState == StateServing || tsv.sm.wantState == StateNotConnected) && !tsv.sm.IsServing() {\n\t\thttp.Error(w, \"500 internal server error: vttablet is not serving\", http.StatusInternalServerError)\n\t\treturn\n\t}\n\tw.Header().Set(\"Content-Length\", strconv.Itoa(len(okMessage)))\n\tw.WriteHeader(http.StatusOK)\n\tw.Write(okMessage)\n}\n\n// Query service health check\n// Returns ok if a query can go all the way to database and back\nfunc (tsv *TabletServer) registerDebugHealthHandler() {\n\ttsv.exporter.HandleFunc(\"/debug/health\", func(w http.ResponseWriter, r *http.Request) {\n\t\tif err := acl.CheckAccessHTTP(r, acl.MONITORING); err != nil {\n\t\t\tacl.SendError(w, err)\n\t\t\treturn\n\t\t}\n\t\tw.Header().Set(\"Content-Type\", \"text/plain\")\n\t\tif err := tsv.IsHealthy(); err != nil {\n\t\t\thttp.Error(w, fmt.Sprintf(\"not ok: %v\", err), http.StatusInternalServerError)","sourceCodeStart":2065,"sourceCodeEnd":2101,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vttablet/tabletserver/tabletserver.go#L2065-L2101","documentation":"This is the HTTP 500 body returned by the vttablet /healthz endpoint when the tablet's state manager says it should be serving (wantState is StateServing or StateNotConnected) but the query service is not actually in the serving state. It signals that the tablet's health check is failing because the underlying query service has not transitioned into serving, typically during startup, shutdown, or after losing connection to MySQL. Load balancers and vtgates use this endpoint to stop routing traffic to the tablet.","triggerScenarios":"An HTTP GET to the vttablet /healthz endpoint while tsv.sm.wantState is StateServing or StateNotConnected and tsv.sm.IsServing() returns false — e.g. the tablet has been told to serve but SetServingType has not completed, or the tabletserver was transitioned to non-serving (e.g. MySQL down, throttle, shutdown) without wantState yet being updated.","commonSituations":"vttablet is still starting up or restarting while a load balancer probes /healthz; MySQL/backup is down so the tabletserver stopped serving; a reparent or tablet restart races with health probes; monitoring scripts polling /healthz during a vttabletd graceful restart.","solutions":["Check the vttablet logs and its /debug/vars (or tablet status page) to see why the query service is not serving — commonly MySQL connectivity — and fix the underlying cause (start/repair mysqld, fix connection params).","Wait for the tablet to finish transitioning to serving (re-run /healthz until it returns 200) or remove the tablet from the load balancer pool until serving.","If the tablet is stuck non-serving, restart vttablet or issue the appropriate SetServing/reparent workflow to transition it back to SERVING.","Verify the tablet's tablet type/state in vtctldclient (gettablet) and make sure a graceful shutdown or reparent is not in progress."],"exampleFix":"// before: prober treats one failing probe as fatal\nif !strings.Contains(resp.Body, \"ok\") {\n    return fmt.Errorf(\"tablet unhealthy\")\n}\n// after: tolerate transient non-serving during startup/restart\nif !strings.Contains(resp.Body, \"ok\") {\n    if errors.Is(err, ErrNotServing) || withinGracePeriod {\n        return ErrRetryLater // allow retries during startup/failover\n    }\n    return fmt.Errorf(\"tablet unhealthy\")\n}","handlingStrategy":"retry","validationCode":"// Check tablet state before routing traffic\nresp, _ := http.Get(\"http://tablet:15001/healthz\")\nif resp == nil || resp.StatusCode != http.StatusOK {\n    // tablet not serving; exclude from pool or retry after backoff\n    return ErrTabletNotReady\n}","typeGuard":null,"tryCatchPattern":"// Go: retry transient not-serving with bounded backoff\nvar lastErr error\nfor i := 0; i < 5; i++ {\n    err := exec.Query(ctx, q)\n    if err == nil {\n        return nil\n    }\n    lastErr = err\n    if strings.Contains(err.Error(), \"vttablet is not serving\") {\n        time.Sleep(backoff(i))\n        continue\n    }\n    break\n}\nreturn lastErr","preventionTips":["Configure load balancers with grace periods so startup/restart probes do not instantly kill routing.","Monitor /healthz trends and alert on sustained non-serving state rather than single probes.","Keep mysqld and vttablet lifecycle managed together (supervised restarts) to avoid non-serving windows.","During planned failovers, drain the tablet from serving before stopping vttablet."],"tags":["vttablet","health-check","serving-state","http"],"backgroundTag":"vttablet-not-serving","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}