What causes "error received from Stream %v" in vitessio/vitess?

Let the player retry — position recovery in _vt.vreplication makes replays safe; check the stream resumes.

error received from Stream %v

Error message

error received from Stream %v

What it means

Thrown by applyEvents when the streaming RPC (BinlogStream) returns a transport/receive error while the player is reading BinlogTransaction responses. Any event already received is dropped (uncommitted) and the error propagates to ApplyBinlogEvents for retry from the last written recovery position.

Source

Thrown at go/vt/binlog/binlogplayer/binlog_player.go:410

			}
			blp.blplStats.ThrottledCounts.Add([]string{"trx", "binlogplayer"}, 1)
			// We don't bother checking for context cancellation here because the
			// sleep will block only up to 1 second. (Usually, backoff is 1s / rate
			// e.g. a rate of 1000 TPS results into a backoff of 1 ms.)
			time.Sleep(backoff)
		}

		// Get the response.
		response, err := stream.Recv()
		// Check context before checking error, because canceled
		// contexts could be wrapped as regular errors.
		select {
		case <-ctx.Done():
			return nil
		default:
		}
		if err != nil {
			return fmt.Errorf("error received from Stream %v", err)
		}

		// process the transaction
		for {
			ok, err = blp.processTransaction(response)
			if err != nil {
				log.Info(fmt.Sprintf("transaction failed: %v", err))
				for _, stmt := range response.Statements {
					log.Info(fmt.Sprintf("statement: %q", stmt.Sql))
				}
				return fmt.Errorf("error in processing binlog event %v", err)
			}
			if ok {
				if !blp.stopPosition.IsZero() {
					if blp.position.AtLeast(blp.stopPosition) {
						msg := "Reached stopping position, done playing logs"
						log.Info(msg)
						if err := blp.setVReplicationState(binlogdatapb.VReplicationWorkflowState_Stopped, msg); err != nil {

View on GitHub (pinned to 01a25a7d17)

Solutions

Let the player retry — position recovery in _vt.vreplication makes replays safe; check the stream resumes.
Verify source tablet health/logs around the failure time for a crash or restart.
Tune gRPC keepalive/LB idle timeouts so long-lived binlog streams are not reaped.
Check network stability (firewall conntrack timeouts) between source and target tablets.

Example fix

// before: default LB may drop idle streams
//   (grpc service config)
// after: add keepalive so streams survive idle periods
//   "grpc.keepalive_time_ms": 30000,
//   "grpc.keepalive_timeout_ms": 10000

Defensive patterns

Strategy: retry

Try / catch

// stream errors are recoverable: retry from stored recovery position
err := binlogplayer.ApplyBinlogEvents(ctx, blp)
if err != nil && strings.Contains(err.Error(), "error received from Stream") {
    log.Warn("binlog stream interrupted; resuming from last recovery position")
    return retryWithBackoff(ctx, func() error {
        return binlogplayer.ApplyBinlogEvents(ctx, blp)
    })
}

Prevention

Set gRPC keepalive so long-lived streams survive LB/proxy idle timeouts
Monitor source tablet restarts and expect stream recovery to kick in
Ensure network path between source and target tablets is stable (conntrack timeouts)
Position recovery makes retry safe — rely on it rather than resyncing

When it happens

Trigger: responseStreamer receiving loop gets err != nil: source tablet closed the stream, gRPC deadline/cancellation upstream, network interruption mid-stream, source binlog server crashed, or context cancelled outside the ctx.Done() fast path.

Common situations: Long-running streams killed by idle-timeout LBs/proxies; source tablet restart during reshard; network blips between regions; gRPC keepalive settings too lax so the connection is silently dropped.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/137cb8ad71e8591a. Report an issue: GitHub.