vitessio/vitess · error

cannot connect to tablet

Error message

cannot connect to tablet %v: %v

What it means

After the explicit healthcheck, WaitForFilteredReplication dials the primary tablet's tabletmanager via tabletconn.GetDialer() to open a StreamHealth connection. If dialing fails, the wait aborts with this message carrying the primary alias and the dial error.

Solutions

1. Confirm the primary vttablet process is up and its grpc port matches the topo record (`vtctldclient GetTablet <alias>`), then re-run.
2. Check TLS settings on both sides (cert, key, root) or dial without TLS if the deployment is plaintext.
3. Test reachability: `nc -zv <tablet-host> <grpc-port>` from the vtctld host.
4. Read the inner dial error for the precise cause (connection refused vs. TLS handshake vs. DNS) and fix that component.

Example fix

// before
vtctldclient WaitForFilteredReplication customer/0 30s
// error: cannot connect to tablet zone1-101: dial tcp 10.0.0.5:16000: connect: connection refused
// after
# restart vttablet on 10.0.0.5 with grpc_port 16000
vtctldclient WaitForFilteredReplication customer/0 30s

// Before waiting, verify the primary's grpc endpoint is reachable
alias, _ := shardPrimaryAlias(keyspace, shard)
tablet, _ := vtctldclientGetTablet(alias)
conn, err := net.DialTimeout("tcp",
    fmt.Sprintf("%s:%d", tablet.MysqlHostname, tablet.PortMap["grpc"]), 2*time.Second)
if err != nil {
    fmt.Printf("primary %s grpc unreachable: %v\n", alias, err)
} else {
    conn.Close()
}

// Go: classify dial errors and retry with backoff
err := wr.WaitForFilteredReplication(ctx, ks, shard, maxDelay)
if err != nil && strings.Contains(err.Error(), "cannot connect to tablet") {
    // transient network issue: retry after verifying tablet process
    time.Sleep(5 * time.Second)
    err = wr.WaitForFilteredReplication(ctx, ks, shard, maxDelay)
}

When it happens

Trigger: Calling WaitForFilteredReplication when the primary tablet's grpc endpoint cannot be dialed — wrong service map, tablet down, DNS/port mismatch, or TLS/auth misconfiguration between vtctld and vttablet.

Common situations: vttablet stopped on the primary; vtctld and vttablet disagree on grpc_port or cert/key configuration; network partition between cells; grpc dialer disabled via service map.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• failed to run explicit healthcheck on tablet
• bad ping result
• cannot dial to server
• cannot execute remote command
• error dialing binlog server

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/8a7da61fb7ada7e3. Report an issue: GitHub.

Appendix: source

Thrown at go/vt/wrangler/split.go:106

	if !shardInfo.HasPrimary() {
		return fmt.Errorf("shard %v/%v has no primary", keyspace, shard)
	}
	alias := shardInfo.PrimaryAlias
	tabletInfo, err := wr.TopoServer().GetTablet(ctx, alias)
	if err != nil {
		return err
	}

	// Always run an explicit healthcheck first to make sure we don't see any outdated values.
	// This is especially true for tests and automation where there is no pause of multiple seconds
	// between commands and the periodic healthcheck did not run again yet.
	if err := wr.TabletManagerClient().RunHealthCheck(ctx, tabletInfo.Tablet); err != nil {
		return fmt.Errorf("failed to run explicit healthcheck on tablet: %v err: %v", tabletInfo, err)
	}

	conn, err := tabletconn.GetDialer()(ctx, tabletInfo.Tablet, grpcclient.FailFast(false))
	if err != nil {
		return fmt.Errorf("cannot connect to tablet %v: %v", alias, err)
	}

	var lastSeenDelay time.Duration
	err = conn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {
		stats := shr.RealtimeStats
		if stats == nil {
			return fmt.Errorf("health record does not include RealtimeStats message. tablet: %v health record: %v", alias, shr)
		}
		if stats.HealthError != "" {
			return fmt.Errorf("tablet is not healthy. tablet: %v health record: %v", alias, shr)
		}
		if stats.BinlogPlayersCount == 0 {
			return fmt.Errorf("no filtered replication running on tablet: %v health record: %v", alias, shr)
		}

		delaySecs := stats.FilteredReplicationLagSeconds
		lastSeenDelay = time.Duration(delaySecs) * time.Second
		if lastSeenDelay < 0 {

View on GitHub (pinned to 01a25a7d17)