vitessio/vitess · error

cannot connect to tablet %v: %v

Error message

cannot connect to tablet %v: %v

What it means

After the explicit healthcheck, WaitForFilteredReplication dials the primary tablet's tabletmanager via tabletconn.GetDialer() to open a StreamHealth connection. If dialing fails, the wait aborts with this message carrying the primary alias and the dial error.

Source

Thrown at go/vt/wrangler/split.go:106

	if !shardInfo.HasPrimary() {
		return fmt.Errorf("shard %v/%v has no primary", keyspace, shard)
	}
	alias := shardInfo.PrimaryAlias
	tabletInfo, err := wr.TopoServer().GetTablet(ctx, alias)
	if err != nil {
		return err
	}

	// Always run an explicit healthcheck first to make sure we don't see any outdated values.
	// This is especially true for tests and automation where there is no pause of multiple seconds
	// between commands and the periodic healthcheck did not run again yet.
	if err := wr.TabletManagerClient().RunHealthCheck(ctx, tabletInfo.Tablet); err != nil {
		return fmt.Errorf("failed to run explicit healthcheck on tablet: %v err: %v", tabletInfo, err)
	}

	conn, err := tabletconn.GetDialer()(ctx, tabletInfo.Tablet, grpcclient.FailFast(false))
	if err != nil {
		return fmt.Errorf("cannot connect to tablet %v: %v", alias, err)
	}

	var lastSeenDelay time.Duration
	err = conn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {
		stats := shr.RealtimeStats
		if stats == nil {
			return fmt.Errorf("health record does not include RealtimeStats message. tablet: %v health record: %v", alias, shr)
		}
		if stats.HealthError != "" {
			return fmt.Errorf("tablet is not healthy. tablet: %v health record: %v", alias, shr)
		}
		if stats.BinlogPlayersCount == 0 {
			return fmt.Errorf("no filtered replication running on tablet: %v health record: %v", alias, shr)
		}

		delaySecs := stats.FilteredReplicationLagSeconds
		lastSeenDelay = time.Duration(delaySecs) * time.Second
		if lastSeenDelay < 0 {

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Confirm the primary vttablet process is up and its grpc port matches the topo record (`vtctldclient GetTablet <alias>`), then re-run.
2. Check TLS settings on both sides (cert, key, root) or dial without TLS if the deployment is plaintext.
3. Test reachability: `nc -zv <tablet-host> <grpc-port>` from the vtctld host.
4. Read the inner dial error for the precise cause (connection refused vs. TLS handshake vs. DNS) and fix that component.

Example fix

// before
vtctldclient WaitForFilteredReplication customer/0 30s
// error: cannot connect to tablet zone1-101: dial tcp 10.0.0.5:16000: connect: connection refused
// after
# restart vttablet on 10.0.0.5 with grpc_port 16000
vtctldclient WaitForFilteredReplication customer/0 30s

// Before waiting, verify the primary's grpc endpoint is reachable
alias, _ := shardPrimaryAlias(keyspace, shard)
tablet, _ := vtctldclientGetTablet(alias)
conn, err := net.DialTimeout("tcp",
    fmt.Sprintf("%s:%d", tablet.MysqlHostname, tablet.PortMap["grpc"]), 2*time.Second)
if err != nil {
    fmt.Printf("primary %s grpc unreachable: %v\n", alias, err)
} else {
    conn.Close()
}

// Go: classify dial errors and retry with backoff
err := wr.WaitForFilteredReplication(ctx, ks, shard, maxDelay)
if err != nil && strings.Contains(err.Error(), "cannot connect to tablet") {
    // transient network issue: retry after verifying tablet process
    time.Sleep(5 * time.Second)
    err = wr.WaitForFilteredReplication(ctx, ks, shard, maxDelay)
}

When it happens

Trigger: Calling WaitForFilteredReplication when the primary tablet's grpc endpoint cannot be dialed — wrong service map, tablet down, DNS/port mismatch, or TLS/auth misconfiguration between vtctld and vttablet.

Common situations: vttablet stopped on the primary; vtctld and vttablet disagree on grpc_port or cert/key configuration; network partition between cells; grpc dialer disabled via service map.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• failed to run explicit healthcheck on tablet: %v err: %v
• gRPC connection wait time exceeded
• error dialing binlog server: %v
• error received from Stream %v
• failed to discover %ss (cluster %s): %w

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/8a7da61fb7ada7e3. Report an issue: GitHub.