{"record":{"id":"8a7da61fb7ada7e3","repo":"vitessio/vitess","slug":"cannot-connect-to-tablet-v-v","errorCode":null,"errorMessage":"cannot connect to tablet %v: %v","messagePattern":"cannot connect to tablet (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/wrangler/split.go","lineNumber":106,"sourceCode":"\tif !shardInfo.HasPrimary() {\n\t\treturn fmt.Errorf(\"shard %v/%v has no primary\", keyspace, shard)\n\t}\n\talias := shardInfo.PrimaryAlias\n\ttabletInfo, err := wr.TopoServer().GetTablet(ctx, alias)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// Always run an explicit healthcheck first to make sure we don't see any outdated values.\n\t// This is especially true for tests and automation where there is no pause of multiple seconds\n\t// between commands and the periodic healthcheck did not run again yet.\n\tif err := wr.TabletManagerClient().RunHealthCheck(ctx, tabletInfo.Tablet); err != nil {\n\t\treturn fmt.Errorf(\"failed to run explicit healthcheck on tablet: %v err: %v\", tabletInfo, err)\n\t}\n\n\tconn, err := tabletconn.GetDialer()(ctx, tabletInfo.Tablet, grpcclient.FailFast(false))\n\tif err != nil {\n\t\treturn fmt.Errorf(\"cannot connect to tablet %v: %v\", alias, err)\n\t}\n\n\tvar lastSeenDelay time.Duration\n\terr = conn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {\n\t\tstats := shr.RealtimeStats\n\t\tif stats == nil {\n\t\t\treturn fmt.Errorf(\"health record does not include RealtimeStats message. tablet: %v health record: %v\", alias, shr)\n\t\t}\n\t\tif stats.HealthError != \"\" {\n\t\t\treturn fmt.Errorf(\"tablet is not healthy. tablet: %v health record: %v\", alias, shr)\n\t\t}\n\t\tif stats.BinlogPlayersCount == 0 {\n\t\t\treturn fmt.Errorf(\"no filtered replication running on tablet: %v health record: %v\", alias, shr)\n\t\t}\n\n\t\tdelaySecs := stats.FilteredReplicationLagSeconds\n\t\tlastSeenDelay = time.Duration(delaySecs) * time.Second\n\t\tif lastSeenDelay < 0 {","sourceCodeStart":88,"sourceCodeEnd":124,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/wrangler/split.go#L88-L124","documentation":"After the explicit healthcheck, WaitForFilteredReplication dials the primary tablet's tabletmanager via tabletconn.GetDialer() to open a StreamHealth connection. If dialing fails, the wait aborts with this message carrying the primary alias and the dial error.","triggerScenarios":"Calling WaitForFilteredReplication when the primary tablet's grpc endpoint cannot be dialed — wrong service map, tablet down, DNS/port mismatch, or TLS/auth misconfiguration between vtctld and vttablet.","commonSituations":"vttablet stopped on the primary; vtctld and vttablet disagree on grpc_port or cert/key configuration; network partition between cells; grpc dialer disabled via service map.","solutions":["Confirm the primary vttablet process is up and its grpc port matches the topo record (`vtctldclient GetTablet <alias>`), then re-run.","Check TLS settings on both sides (cert, key, root) or dial without TLS if the deployment is plaintext.","Test reachability: `nc -zv <tablet-host> <grpc-port>` from the vtctld host.","Read the inner dial error for the precise cause (connection refused vs. TLS handshake vs. DNS) and fix that component."],"exampleFix":"// before\nvtctldclient WaitForFilteredReplication customer/0 30s\n// error: cannot connect to tablet zone1-101: dial tcp 10.0.0.5:16000: connect: connection refused\n// after\n# restart vttablet on 10.0.0.5 with grpc_port 16000\nvtctldclient WaitForFilteredReplication customer/0 30s","handlingStrategy":"retry","validationCode":"// Before waiting, verify the primary's grpc endpoint is reachable\nalias, _ := shardPrimaryAlias(keyspace, shard)\ntablet, _ := vtctldclientGetTablet(alias)\nconn, err := net.DialTimeout(\"tcp\",\n    fmt.Sprintf(\"%s:%d\", tablet.MysqlHostname, tablet.PortMap[\"grpc\"]), 2*time.Second)\nif err != nil {\n    fmt.Printf(\"primary %s grpc unreachable: %v\\n\", alias, err)\n} else {\n    conn.Close()\n}","typeGuard":null,"tryCatchPattern":"// Go: classify dial errors and retry with backoff\nerr := wr.WaitForFilteredReplication(ctx, ks, shard, maxDelay)\nif err != nil && strings.Contains(err.Error(), \"cannot connect to tablet\") {\n    // transient network issue: retry after verifying tablet process\n    time.Sleep(5 * time.Second)\n    err = wr.WaitForFilteredReplication(ctx, ks, shard, maxDelay)\n}","preventionTips":["Keep vttablet grpc ports consistent between topo records and process flags","Align TLS cert/key/root configuration across vtctld and vttablets","Pre-check grpc port reachability from the vtctld host in runbooks","Watch for network partitions in multi-cell deployments before reshard cutover"],"tags":["network","grpc","tabletmanager","filtered-replication"],"backgroundTag":"connection-refused","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}