{"record":{"id":"3298102edec38edf","repo":"vitessio/vitess","slug":"vttablet-error-v","errorCode":null,"errorMessage":"vttablet error: %v","messagePattern":"vttablet error: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/discovery/tablet_health_check.go","lineNumber":177,"sourceCode":"\n// processResponse reads one health check response, and updates health\nfunc (thc *tabletHealthCheck) processResponse(hc *HealthCheckImpl, shr *query.StreamHealthResponse) error {\n\tselect {\n\tcase <-thc.ctx.Done():\n\t\treturn thc.ctx.Err()\n\tdefault:\n\t}\n\n\t// Check for invalid data, better than panicking.\n\tif shr.Target == nil || shr.RealtimeStats == nil {\n\t\treturn fmt.Errorf(\"health stats is not valid: %v\", shr)\n\t}\n\n\t// an app-level error from tablet, force serving state.\n\tvar healthErr error\n\tserving := shr.Serving\n\tif shr.RealtimeStats.HealthError != \"\" {\n\t\thealthErr = fmt.Errorf(\"vttablet error: %v\", shr.RealtimeStats.HealthError)\n\t\tserving = false\n\t}\n\n\tif shr.TabletAlias != nil && !proto.Equal(shr.TabletAlias, thc.Tablet.Alias) {\n\t\t// TabletAlias change means that the host:port has been taken over by another tablet\n\t\t// We cancel / exit the healthcheck for this tablet right away\n\t\t// With the next topo refresh we will get a new tablet with the new host/port\n\t\treturn vterrors.New(vtrpc.Code_FAILED_PRECONDITION, fmt.Sprintf(\"health stats mismatch, tablet %+v alias does not match response alias %v\", thc.Tablet, shr.TabletAlias))\n\t}\n\n\tprevTarget := thc.Target\n\t// check whether this is a trivial update so as to update healthy map\n\ttrivialUpdate := thc.LastError == nil && thc.Serving && shr.RealtimeStats.HealthError == \"\" && shr.Serving &&\n\t\tprevTarget.TabletType != topodata.TabletType_PRIMARY && prevTarget.TabletType == shr.Target.TabletType && thc.isTrivialReplagChange(shr.RealtimeStats)\n\tthc.lastResponseTimestamp = time.Now()\n\tthc.Target = shr.Target\n\tthc.PrimaryTermStartTime = shr.PrimaryTermStartTimestamp\n\tthc.Stats = shr.RealtimeStats","sourceCodeStart":159,"sourceCodeEnd":195,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/discovery/tablet_health_check.go#L159-L195","documentation":"When a vttablet reports a non-empty HealthError in its RealtimeStats, the healthcheck converts it into this 'vttablet error' and forces the tablet's serving state to false. It reflects an application-level health failure reported by the tablet itself (e.g. MySQL replication lag or liveness check failing), not a transport problem.","triggerScenarios":"processResponse receives a StreamHealthResponse whose RealtimeStats.HealthError is non-empty — vttablet's health check failed (e.g. MySQL down, replication lag above threshold, tablet overwhelmed).","commonSituations":"MySQL replica lagging beyond the lameduck/hi-jack threshold; mysqld restart on the tablet host; disk full or table locked causing health queries to fail; tablet freshly started before MySQL is ready.","solutions":["Read the wrapped %v message — it contains vttablet's actual HealthError text and diagnose that condition.","Check replication status on the tablet's MySQL (SHOW REPLICA STATUS) for lag or broken replication.","Verify mysqld is running and responsive on the tablet host; restart if needed.","Once the underlying health condition clears, the tablet will resume serving automatically on the next healthy StreamHealthResponse."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"hd, err := hc.GetTabletHealthByAlias(ctx, alias)\nif err != nil && strings.Contains(err.Error(), \"vttablet error\") {\n\t// tablet reported unhealthy; retry or route traffic elsewhere\n\treturn routeAround(alias)\n}","preventionTips":["Alert on vttablet HealthError strings to catch MySQL/replication issues early","Keep replication lag thresholds tuned to your workload","Monitor mysqld availability on tablet hosts"],"tags":["healthcheck","vttablet","replication"],"backgroundTag":"tablet-health-error","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}