{"record":{"id":"0f3dbcc64b225ed4","repo":"vitessio/vitess","slug":"not-ok-v","errorCode":null,"errorMessage":"not ok: %v","messagePattern":"not ok: (.+?)","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"go/vt/vttablet/tabletserver/tabletserver.go","lineNumber":2101,"sourceCode":"\t\thttp.Error(w, \"500 internal server error: vttablet is not serving\", http.StatusInternalServerError)\n\t\treturn\n\t}\n\tw.Header().Set(\"Content-Length\", strconv.Itoa(len(okMessage)))\n\tw.WriteHeader(http.StatusOK)\n\tw.Write(okMessage)\n}\n\n// Query service health check\n// Returns ok if a query can go all the way to database and back\nfunc (tsv *TabletServer) registerDebugHealthHandler() {\n\ttsv.exporter.HandleFunc(\"/debug/health\", func(w http.ResponseWriter, r *http.Request) {\n\t\tif err := acl.CheckAccessHTTP(r, acl.MONITORING); err != nil {\n\t\t\tacl.SendError(w, err)\n\t\t\treturn\n\t\t}\n\t\tw.Header().Set(\"Content-Type\", \"text/plain\")\n\t\tif err := tsv.IsHealthy(); err != nil {\n\t\t\thttp.Error(w, fmt.Sprintf(\"not ok: %v\", err), http.StatusInternalServerError)\n\t\t\treturn\n\t\t}\n\t\tw.Write([]byte(\"ok\"))\n\t})\n}\n\nfunc (tsv *TabletServer) registerQueryzHandler() {\n\ttsv.exporter.HandleFunc(\"/queryz\", func(w http.ResponseWriter, r *http.Request) {\n\t\tqueryzHandler(tsv.qe, w, r)\n\t})\n}\n\nfunc (tsv *TabletServer) registerQuerylogzHandler() {\n\ttsv.exporter.HandleFunc(\"/querylogz\", func(w http.ResponseWriter, r *http.Request) {\n\t\tch := tabletenv.StatsLogger.Subscribe(\"querylogz\")\n\t\tdefer tabletenv.StatsLogger.Unsubscribe(ch)\n\t\tquerylogzHandler(ch, w, r, tsv.env.Parser())\n\t})","sourceCodeStart":2083,"sourceCodeEnd":2119,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vttablet/tabletserver/tabletserver.go#L2083-L2119","documentation":"This is the HTTP 500 body returned by the vttablet /debug/health endpoint when tsv.IsHealthy() returns an error. Unlike /healthz, this check actually verifies the tablet can run a query (usually a ping against MySQL), so the error message wraps the underlying health failure (e.g. connection to MySQL failed, query timeout, tablet not serving). It indicates the tablet cannot currently process queries end-to-end.","triggerScenarios":"An HTTP GET to /debug/health (with acl MONITORING access) while tsv.IsHealthy() fails — typically because the MySQL backend is unreachable, the health check query times out, the tabletserver is not serving, or transactions/queries are in a bad state (e.g. during shutdown).","commonSituations":"mysqld down or restarted; network partition between vttablet and MySQL; MySQL overloaded so the health query exceeds the timeout; monitoring/consul health checks flipping the tablet out of the serving set; acl misconfiguration blocking the request (returns a different acl error though).","solutions":["Read the error detail after 'not ok:' in the response — it names the underlying health failure — and fix that cause (restore MySQL connectivity, address replication lag/timeout).","Check vttablet logs around the failing health check for the full IsHealthy error and any MySQL errors.","Confirm mysqld is up (mysqlctl status) and that the tablet can reconnect; restart vttablet if it is stuck in a failed state.","If this happens under load only, increase the health check tolerance or scale MySQL resources; if it persists, take the tablet out of rotation and investigate."],"exampleFix":"// before: ignoring the detailed error in monitoring\nif !strings.HasSuffix(body, \"ok\") { alert(\"tablet down\") }\n// after: parse and surface the underlying cause\nif !strings.HasSuffix(body, \"ok\") {\n    cause := strings.TrimPrefix(body, \"not ok: \")\n    alert(\"tablet down: \" + cause) // e.g. 'connection to mysql failed'\n}","handlingStrategy":"type-guard","validationCode":"// Probe deep health before sending queries\ntype HealthChecker interface { IsHealthy() error }\nif err := h.IsHealthy(); err != nil {\n    return fmt.Errorf(\"tablet deep health failed: %w\", err)\n}","typeGuard":"func isDeepHealthError(body string) bool {\n    return strings.HasPrefix(body, \"not ok: \")\n}\n// if isDeepHealthError(resp.Body) { parse the cause after 'not ok: ' }","tryCatchPattern":"body, err := io.ReadAll(resp.Body)\nif resp.StatusCode != http.StatusOK && strings.HasPrefix(string(body), \"not ok: \") {\n    cause := strings.TrimPrefix(string(body), \"not ok: \")\n    return fmt.Errorf(\"deep health check failed: %s\", cause)\n}","preventionTips":["Alert on the specific cause after 'not ok:' (mysql connection, timeout) instead of just the 500 status.","Verify MySQL connectivity and replication health proactively so deep health checks stay green.","Ensure MONITORING acl is configured for health probes to avoid confusing acl errors with health failures.","Use /healthz (state-only) and /debug/health (deep) together to distinguish serving-state issues from backend issues."],"tags":["vttablet","health-check","mysql","monitoring"],"backgroundTag":"health-check-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}