{"record":{"id":"e2dccab6b2f36648","repo":"vitessio/vitess","slug":"error-dialing-binlog-server-v","errorCode":null,"errorMessage":"error dialing binlog server: %v","messagePattern":"error dialing binlog server: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/binlog/binlogplayer/binlog_player.go","lineNumber":344,"sourceCode":"\t\t\tlog.Error(msg)\n\t\t\tif err := blp.setVReplicationState(binlogdatapb.VReplicationWorkflowState_Stopped, msg); err != nil {\n\t\t\t\tlog.Error(fmt.Sprintf(\"Error writing stop state: %v\", err))\n\t\t\t}\n\t\t\t// Don't return an error. Otherwise, it will keep retrying.\n\t\t\treturn nil\n\t\tdefault:\n\t\t\tlog.Info(fmt.Sprintf(\"Will stop player when reaching %v\", blp.stopPosition))\n\t\t}\n\t}\n\n\tclientFactory, ok := clientFactories[binlogPlayerProtocol]\n\tif !ok {\n\t\treturn fmt.Errorf(\"no binlog player client factory named %v\", binlogPlayerProtocol)\n\t}\n\tblplClient := clientFactory()\n\terr = blplClient.Dial(ctx, blp.tablet)\n\tif err != nil {\n\t\terr := fmt.Errorf(\"error dialing binlog server: %v\", err)\n\t\tlog.Error(fmt.Sprint(err))\n\t\treturn err\n\t}\n\tdefer blplClient.Close()\n\n\t// Get the current charset of our connection, so we can ask the stream server\n\t// to check that they match. The streamer will also only send per-statement\n\t// charset data if that statement's charset is different from what we specify.\n\tif dbClient, ok := blp.dbClient.(*dbClientImpl); ok {\n\t\tblp.defaultCharset, err = mysql.GetCharset(dbClient.dbConn)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"can't get charset to request binlog stream: %v\", err)\n\t\t}\n\t\tlog.Info(fmt.Sprintf(\"original charset: %v\", blp.defaultCharset))\n\t\tblp.currentCharset = blp.defaultCharset\n\t\t// Restore original charset when we're done.\n\t\tdefer func() {\n\t\t\t// If the connection has been closed, there's no need to restore","sourceCodeStart":326,"sourceCodeEnd":362,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/binlog/binlogplayer/binlog_player.go#L326-L362","documentation":"Thrown by applyEvents when the BinlogPlayer's client factory returns a client but the client's Dial() call to the binlog server (vstream/vreplication source tablet) fails. The original dial error is wrapped and logged, then returned so ApplyBinlogEvents can retry with backoff. It means the player could not establish an RPC connection to the source binlog server at all.","triggerScenarios":"blplClient.Dial(ctx, blp.tablet) returns an error: source tablet is down, wrong tablet alias/address, network partition, source tablet serving type changed, TLS/auth mismatch, or the binlog server process is not listening on its grpc port.","commonSituations":"VReplication streams break during tablet restarts, failovers, pruning of source tablets, k8s pod rescheduling, firewall/security-group changes, or misconfigured vtctld spec pointing at a retired tablet.","solutions":["Verify the source tablet is running and serving: check `vtctldclient GetTablets` for the alias in blp.tablet and that its binlog/vreplication service is up.","Check network reachability (grpc port) from the player's host to the source tablet: `nc -vz <tablet-host> <grpc-port>`.","Confirm the binlog player protocol/factory name is registered (clientFactory lookup passed, so protocol is fine — focus on connectivity).","Restart or re-create the vreplication stream (e.g. `vtctldclient VReplicationExec ... / reshard restart`) so it re-resolves the current source tablet.","Inspect tablet logs for TLS/certificate or auth errors if using mutual TLS between vttablets."],"exampleFix":"// before: stale tablet alias after failover\n//   err = blpClient.Dial(ctx, blp.tablet) // dial to dead tablet\n// after: re-resolve source before dialing (ops-level fix)\n//   vtctldclient VReplicationExec <workflow> update-set-source ...\n//   (stream restarts and dials the current serving tablet)","handlingStrategy":"retry","validationCode":"// before starting the stream, verify the source is dialable\nfunc validateSource(addr string, port int) error {\n    conn, err := net.DialTimeout(\"tcp\", fmt.Sprintf(\"%s:%d\", addr, port), 5*time.Second)\n    if err != nil {\n        return fmt.Errorf(\"source binlog server %s:%d unreachable: %w\", addr, port, err)\n    }\n    conn.Close()\n    return nil\n}","typeGuard":null,"tryCatchPattern":"// retry dial with backoff; position recovery makes replays safe\nfor attempt := 0; attempt < 5; attempt++ {\n    err := blplClient.Dial(ctx, blp.tablet)\n    if err == nil {\n        break\n    }\n    log.Error(fmt.Sprintf(\"error dialing binlog server: %v (attempt %d)\", err, attempt+1))\n    select {\n    case <-ctx.Done():\n        return ctx.Err()\n    case <-time.After(backoff(attempt)):\n    }\n}","preventionTips":["Monitor tablet health and ensure vreplication source tablets stay serving for the life of the workflow","Re-resolve source tablet aliases after failovers instead of pinning addresses","Keep gRPC ports open between all tablet pairs (check firewall/security groups)","Alert on vreplication state='Error' in _vt.vreplication"],"tags":["network","grpc","dial","binlog-server","vreplication"],"backgroundTag":"connection-refused","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}