vitessio/vitess · error

error dialing binlog server: %v

Error message

error dialing binlog server: %v

What it means

Thrown by applyEvents when the BinlogPlayer's client factory returns a client but the client's Dial() call to the binlog server (vstream/vreplication source tablet) fails. The original dial error is wrapped and logged, then returned so ApplyBinlogEvents can retry with backoff. It means the player could not establish an RPC connection to the source binlog server at all.

Source

Thrown at go/vt/binlog/binlogplayer/binlog_player.go:344

			log.Error(msg)
			if err := blp.setVReplicationState(binlogdatapb.VReplicationWorkflowState_Stopped, msg); err != nil {
				log.Error(fmt.Sprintf("Error writing stop state: %v", err))
			}
			// Don't return an error. Otherwise, it will keep retrying.
			return nil
		default:
			log.Info(fmt.Sprintf("Will stop player when reaching %v", blp.stopPosition))
		}
	}

	clientFactory, ok := clientFactories[binlogPlayerProtocol]
	if !ok {
		return fmt.Errorf("no binlog player client factory named %v", binlogPlayerProtocol)
	}
	blplClient := clientFactory()
	err = blplClient.Dial(ctx, blp.tablet)
	if err != nil {
		err := fmt.Errorf("error dialing binlog server: %v", err)
		log.Error(fmt.Sprint(err))
		return err
	}
	defer blplClient.Close()

	// Get the current charset of our connection, so we can ask the stream server
	// to check that they match. The streamer will also only send per-statement
	// charset data if that statement's charset is different from what we specify.
	if dbClient, ok := blp.dbClient.(*dbClientImpl); ok {
		blp.defaultCharset, err = mysql.GetCharset(dbClient.dbConn)
		if err != nil {
			return fmt.Errorf("can't get charset to request binlog stream: %v", err)
		}
		log.Info(fmt.Sprintf("original charset: %v", blp.defaultCharset))
		blp.currentCharset = blp.defaultCharset
		// Restore original charset when we're done.
		defer func() {
			// If the connection has been closed, there's no need to restore

View on GitHub (pinned to 01a25a7d17)

Solutions

  1. Verify the source tablet is running and serving: check `vtctldclient GetTablets` for the alias in blp.tablet and that its binlog/vreplication service is up.
  2. Check network reachability (grpc port) from the player's host to the source tablet: `nc -vz <tablet-host> <grpc-port>`.
  3. Confirm the binlog player protocol/factory name is registered (clientFactory lookup passed, so protocol is fine — focus on connectivity).
  4. Restart or re-create the vreplication stream (e.g. `vtctldclient VReplicationExec ... / reshard restart`) so it re-resolves the current source tablet.
  5. Inspect tablet logs for TLS/certificate or auth errors if using mutual TLS between vttablets.

Example fix

// before: stale tablet alias after failover
//   err = blpClient.Dial(ctx, blp.tablet) // dial to dead tablet
// after: re-resolve source before dialing (ops-level fix)
//   vtctldclient VReplicationExec <workflow> update-set-source ...
//   (stream restarts and dials the current serving tablet)
Defensive patterns

Strategy: retry

Validate before calling

// before starting the stream, verify the source is dialable
func validateSource(addr string, port int) error {
    conn, err := net.DialTimeout("tcp", fmt.Sprintf("%s:%d", addr, port), 5*time.Second)
    if err != nil {
        return fmt.Errorf("source binlog server %s:%d unreachable: %w", addr, port, err)
    }
    conn.Close()
    return nil
}

Try / catch

// retry dial with backoff; position recovery makes replays safe
for attempt := 0; attempt < 5; attempt++ {
    err := blplClient.Dial(ctx, blp.tablet)
    if err == nil {
        break
    }
    log.Error(fmt.Sprintf("error dialing binlog server: %v (attempt %d)", err, attempt+1))
    select {
    case <-ctx.Done():
        return ctx.Err()
    case <-time.After(backoff(attempt)):
    }
}

Prevention

When it happens

Trigger: blplClient.Dial(ctx, blp.tablet) returns an error: source tablet is down, wrong tablet alias/address, network partition, source tablet serving type changed, TLS/auth mismatch, or the binlog server process is not listening on its grpc port.

Common situations: VReplication streams break during tablet restarts, failovers, pruning of source tablets, k8s pod rescheduling, firewall/security-group changes, or misconfigured vtctld spec pointing at a retired tablet.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/e2dccab6b2f36648. Report an issue: GitHub.