vitessio/vitess · error

failed to run explicit healthcheck on tablet: %v err: %v

Error message

failed to run explicit healthcheck on tablet: %v err: %v

What it means

Before streaming health, WaitForFilteredReplication forces an on-demand health check on the shard's primary tablet so it doesn't read stale lag values (periodic healthchecks may not have run recently, especially in tests/automation). If the RunHealthCheck RPC to the tabletmanager fails, the wait aborts with this wrapped error.

Source

Thrown at go/vt/wrangler/split.go:101

		return err
	}
	if len(shardInfo.SourceShards) == 0 {
		return fmt.Errorf("shard %v/%v has no source shard", keyspace, shard)
	}
	if !shardInfo.HasPrimary() {
		return fmt.Errorf("shard %v/%v has no primary", keyspace, shard)
	}
	alias := shardInfo.PrimaryAlias
	tabletInfo, err := wr.TopoServer().GetTablet(ctx, alias)
	if err != nil {
		return err
	}

	// Always run an explicit healthcheck first to make sure we don't see any outdated values.
	// This is especially true for tests and automation where there is no pause of multiple seconds
	// between commands and the periodic healthcheck did not run again yet.
	if err := wr.TabletManagerClient().RunHealthCheck(ctx, tabletInfo.Tablet); err != nil {
		return fmt.Errorf("failed to run explicit healthcheck on tablet: %v err: %v", tabletInfo, err)
	}

	conn, err := tabletconn.GetDialer()(ctx, tabletInfo.Tablet, grpcclient.FailFast(false))
	if err != nil {
		return fmt.Errorf("cannot connect to tablet %v: %v", alias, err)
	}

	var lastSeenDelay time.Duration
	err = conn.StreamHealth(ctx, func(shr *querypb.StreamHealthResponse) error {
		stats := shr.RealtimeStats
		if stats == nil {
			return fmt.Errorf("health record does not include RealtimeStats message. tablet: %v health record: %v", alias, shr)
		}
		if stats.HealthError != "" {
			return fmt.Errorf("tablet is not healthy. tablet: %v health record: %v", alias, shr)
		}
		if stats.BinlogPlayersCount == 0 {
			return fmt.Errorf("no filtered replication running on tablet: %v health record: %v", alias, shr)

View on GitHub (pinned to 01a25a7d17)

Solutions

  1. Check the primary tablet is running and its grpc port is reachable (`grpc_port` in the tablet record); restart vttablet if needed.
  2. Read the inner `err` in the message for the RPC failure cause (timeout, connection refused, TLS) and fix accordingly.
  3. Verify the tablet's health with `vtctldclient GetTablet <alias>` and `vtctldclient RunHealthCheck <alias>`, then re-run the wait.
  4. Increase the command's context timeout if the tablet was merely slow to respond.

Example fix

// before
vtctldclient WaitForFilteredReplication customer/0 30s
// error: ... err: rpc error: connection refused
// after
ps aux | grep vttablet                 # ensure vttablet for zone1-101 is up
vtctldclient RunHealthCheck zone1-101
vtctldclient WaitForFilteredReplication customer/0 30s
Defensive patterns

Strategy: retry

Validate before calling

// Before waiting, confirm the primary tablet is healthy
alias, _ := shardPrimaryAlias(keyspace, shard)
tablet, _ := vtctldclientGetTablet(alias)
if tablet.State != "SERVING" {
    fmt.Printf("primary %s not serving; fix vttablet first\n", alias)
}

Try / catch

// Go: retry transient RunHealthCheck failures with backoff
for attempt := 0; attempt < 3; attempt++ {
    err := wr.WaitForFilteredReplication(ctx, ks, shard, maxDelay)
    if err == nil {
        return nil
    }
    if strings.Contains(err.Error(), "failed to run explicit healthcheck") {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(time.Duration(attempt+1) * 2 * time.Second):
        }
        continue
    }
    return err
}
return err

Prevention

When it happens

Trigger: Calling WaitForFilteredReplication when the primary tablet is unreachable, its tabletmanager is down/restarting, or the RPC times out.

Common situations: Primary tablet process crashed or vttablet not listening on its grpc port; firewall blocking vtctld→vttablet traffic; tablet restarting during automated reshard tests.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/2baa84fd676cd1f9. Report an issue: GitHub.