{"record":{"id":"2baa84fd676cd1f9","repo":"vitessio/vitess","slug":"failed-to-run-explicit-healthcheck-on-tablet-v-e","errorCode":null,"errorMessage":"failed to run explicit healthcheck on tablet: %v err: %v","messagePattern":"failed to run explicit healthcheck on tablet: (.+?) err: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/wrangler/split.go","lineNumber":101,"sourceCode":"\t\treturn err\n\t}\n\tif len(shardInfo.SourceShards) == 0 {\n\t\treturn fmt.Errorf(\"shard %v/%v has no source shard\", keyspace, shard)\n\t}\n\tif !shardInfo.HasPrimary() {\n\t\treturn fmt.Errorf(\"shard %v/%v has no primary\", keyspace, shard)\n\t}\n\talias := shardInfo.PrimaryAlias\n\ttabletInfo, err := wr.TopoServer().GetTablet(ctx, alias)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// Always run an explicit healthcheck first to make sure we don't see any outdated values.\n\t// This is especially true for tests and automation where there is no pause of multiple seconds\n\t// between commands and the periodic healthcheck did not run again yet.\n\tif err := wr.TabletManagerClient().RunHealthCheck(ctx, tabletInfo.Tablet); err != nil {\n\t\treturn fmt.Errorf(\"failed to run explicit healthcheck on tablet: %v err: %v\", tabletInfo, err)\n\t}\n\n\tconn, err := tabletconn.GetDialer()(ctx, tabletInfo.Tablet, grpcclient.FailFast(false))\n\tif err != nil {\n\t\treturn fmt.Errorf(\"cannot connect to tablet %v: %v\", alias, err)\n\t}\n\n\tvar lastSeenDelay time.Duration\n\terr = conn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {\n\t\tstats := shr.RealtimeStats\n\t\tif stats == nil {\n\t\t\treturn fmt.Errorf(\"health record does not include RealtimeStats message. tablet: %v health record: %v\", alias, shr)\n\t\t}\n\t\tif stats.HealthError != \"\" {\n\t\t\treturn fmt.Errorf(\"tablet is not healthy. tablet: %v health record: %v\", alias, shr)\n\t\t}\n\t\tif stats.BinlogPlayersCount == 0 {\n\t\t\treturn fmt.Errorf(\"no filtered replication running on tablet: %v health record: %v\", alias, shr)","sourceCodeStart":83,"sourceCodeEnd":119,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/wrangler/split.go#L83-L119","documentation":"Before streaming health, WaitForFilteredReplication forces an on-demand health check on the shard's primary tablet so it doesn't read stale lag values (periodic healthchecks may not have run recently, especially in tests/automation). If the RunHealthCheck RPC to the tabletmanager fails, the wait aborts with this wrapped error.","triggerScenarios":"Calling WaitForFilteredReplication when the primary tablet is unreachable, its tabletmanager is down/restarting, or the RPC times out.","commonSituations":"Primary tablet process crashed or vttablet not listening on its grpc port; firewall blocking vtctld→vttablet traffic; tablet restarting during automated reshard tests.","solutions":["Check the primary tablet is running and its grpc port is reachable (`grpc_port` in the tablet record); restart vttablet if needed.","Read the inner `err` in the message for the RPC failure cause (timeout, connection refused, TLS) and fix accordingly.","Verify the tablet's health with `vtctldclient GetTablet <alias>` and `vtctldclient RunHealthCheck <alias>`, then re-run the wait.","Increase the command's context timeout if the tablet was merely slow to respond."],"exampleFix":"// before\nvtctldclient WaitForFilteredReplication customer/0 30s\n// error: ... err: rpc error: connection refused\n// after\nps aux | grep vttablet                 # ensure vttablet for zone1-101 is up\nvtctldclient RunHealthCheck zone1-101\nvtctldclient WaitForFilteredReplication customer/0 30s","handlingStrategy":"retry","validationCode":"// Before waiting, confirm the primary tablet is healthy\nalias, _ := shardPrimaryAlias(keyspace, shard)\ntablet, _ := vtctldclientGetTablet(alias)\nif tablet.State != \"SERVING\" {\n    fmt.Printf(\"primary %s not serving; fix vttablet first\\n\", alias)\n}","typeGuard":null,"tryCatchPattern":"// Go: retry transient RunHealthCheck failures with backoff\nfor attempt := 0; attempt < 3; attempt++ {\n    err := wr.WaitForFilteredReplication(ctx, ks, shard, maxDelay)\n    if err == nil {\n        return nil\n    }\n    if strings.Contains(err.Error(), \"failed to run explicit healthcheck\") {\n        select {\n        case <-ctx.Done():\n            return ctx.Err()\n        case <-time.After(time.Duration(attempt+1) * 2 * time.Second):\n        }\n        continue\n    }\n    return err\n}\nreturn err","preventionTips":["Monitor vttablet process/liveness on primaries used in reshard workflows","Ensure vtctld can reach every tablet's grpc port","Allow tablet restarts to finish before invoking wait commands in automation"],"tags":["healthcheck","tabletmanager","network","filtered-replication"],"backgroundTag":"healthcheck-rpc-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}