{"record":{"id":"137cb8ad71e8591a","repo":"vitessio/vitess","slug":"error-received-from-stream-v","errorCode":null,"errorMessage":"error received from Stream %v","messagePattern":"error received from Stream (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/binlog/binlogplayer/binlog_player.go","lineNumber":410,"sourceCode":"\t\t\t}\n\t\t\tblp.blplStats.ThrottledCounts.Add([]string{\"trx\", \"binlogplayer\"}, 1)\n\t\t\t// We don't bother checking for context cancellation here because the\n\t\t\t// sleep will block only up to 1 second. (Usually, backoff is 1s / rate\n\t\t\t// e.g. a rate of 1000 TPS results into a backoff of 1 ms.)\n\t\t\ttime.Sleep(backoff)\n\t\t}\n\n\t\t// Get the response.\n\t\tresponse, err := stream.Recv()\n\t\t// Check context before checking error, because canceled\n\t\t// contexts could be wrapped as regular errors.\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn nil\n\t\tdefault:\n\t\t}\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"error received from Stream %v\", err)\n\t\t}\n\n\t\t// process the transaction\n\t\tfor {\n\t\t\tok, err = blp.processTransaction(response)\n\t\t\tif err != nil {\n\t\t\t\tlog.Info(fmt.Sprintf(\"transaction failed: %v\", err))\n\t\t\t\tfor _, stmt := range response.Statements {\n\t\t\t\t\tlog.Info(fmt.Sprintf(\"statement: %q\", stmt.Sql))\n\t\t\t\t}\n\t\t\t\treturn fmt.Errorf(\"error in processing binlog event %v\", err)\n\t\t\t}\n\t\t\tif ok {\n\t\t\t\tif !blp.stopPosition.IsZero() {\n\t\t\t\t\tif blp.position.AtLeast(blp.stopPosition) {\n\t\t\t\t\t\tmsg := \"Reached stopping position, done playing logs\"\n\t\t\t\t\t\tlog.Info(msg)\n\t\t\t\t\t\tif err := blp.setVReplicationState(binlogdatapb.VReplicationWorkflowState_Stopped, msg); err != nil {","sourceCodeStart":392,"sourceCodeEnd":428,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/binlog/binlogplayer/binlog_player.go#L392-L428","documentation":"Thrown by applyEvents when the streaming RPC (BinlogStream) returns a transport/receive error while the player is reading BinlogTransaction responses. Any event already received is dropped (uncommitted) and the error propagates to ApplyBinlogEvents for retry from the last written recovery position.","triggerScenarios":"responseStreamer receiving loop gets err != nil: source tablet closed the stream, gRPC deadline/cancellation upstream, network interruption mid-stream, source binlog server crashed, or context cancelled outside the ctx.Done() fast path.","commonSituations":"Long-running streams killed by idle-timeout LBs/proxies; source tablet restart during reshard; network blips between regions; gRPC keepalive settings too lax so the connection is silently dropped.","solutions":["Let the player retry — position recovery in _vt.vreplication makes replays safe; check the stream resumes.","Verify source tablet health/logs around the failure time for a crash or restart.","Tune gRPC keepalive/LB idle timeouts so long-lived binlog streams are not reaped.","Check network stability (firewall conntrack timeouts) between source and target tablets."],"exampleFix":"// before: default LB may drop idle streams\n//   (grpc service config)\n// after: add keepalive so streams survive idle periods\n//   \"grpc.keepalive_time_ms\": 30000,\n//   \"grpc.keepalive_timeout_ms\": 10000","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// stream errors are recoverable: retry from stored recovery position\nerr := binlogplayer.ApplyBinlogEvents(ctx, blp)\nif err != nil && strings.Contains(err.Error(), \"error received from Stream\") {\n    log.Warn(\"binlog stream interrupted; resuming from last recovery position\")\n    return retryWithBackoff(ctx, func() error {\n        return binlogplayer.ApplyBinlogEvents(ctx, blp)\n    })\n}","preventionTips":["Set gRPC keepalive so long-lived streams survive LB/proxy idle timeouts","Monitor source tablet restarts and expect stream recovery to kick in","Ensure network path between source and target tablets is stable (conntrack timeouts)","Position recovery makes retry safe — rely on it rather than resyncing"],"tags":["grpc","stream","network","binlog-player","vreplication"],"backgroundTag":"stream-disconnected","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}