stream error @ (including the GTID we failed to process)

Error message

stream error @ (including the GTID we failed to process) %v: %v

What it means

In binlog_streamer.go, when streaming ends with an error other than ErrBinlogUnavailable, the deferred handler wraps the error with the last known position (stopPos) as "stream error @ (including the GTID we failed to process) <position>: <original>". The position marker tells operators exactly which GTID/binlog position the streamer had reached (and could not get past) when the failure occurred, since the stop position is updated as events are successfully processed.

Solutions

Unwrap the error: the %v suffix carries the root cause; fix that first (reconnect, corruption, permissions, etc.).
Use the @ position in the message to resume streaming from (or inspect the binlog at) the exact GTID that failed.
If the primary failed over, reparent and restart the stream — the streamer will reconnect from the recorded position.
If the failing GTID corresponds to a corrupt/unavailable binlog file, restore it from backup or use a vreplication copy/reshard to re-sync.
Check MySQL error logs and binlog integrity (mysqlbinlog) at the reported position.

Example fix

// before: treats all stream failures identically
if err != nil {
	return err
}
// after: distinguish expected unavailability from real stream failures
if errors.Is(err, ErrBinlogUnavailable) {
	// wait/refresh binlog sources, then retry
} else if err != nil {
	log.Error("binlog stream failed",
		slog.Any("position", stopPos), slog.Any("error", err))
	return err
}

Defensive patterns

Strategy: try-catch

Validate before calling

// before streaming, confirm the starting position exists in the primary's binlogs
qr, _ := conn.ExecuteFetch("SHOW BINARY LOG STATUS", 1, false)
_ = qr // compare startPos against current file list/GTID set

Try / catch

err := bls.Stream(ctx)
if err != nil {
	var stopPos replication.Position
	if m := regexp.MustCompile(`stream error @ (\([^)]*\))?[^:]*: `).FindStringSubmatch(err.Error()); m != nil {
		_ = m // resume from this position after fixing root cause
	}
	if strings.Contains(err.Error(), "ErrBinlogUnavailable") {
		// wait for binlogs / refresh sources
	}
}

Prevention

Always log and persist the stream position on failure so restarts resume at the right GTID.
Distinguish ErrBinlogUnavailable from other failures before retrying.
Monitor primaries for failover/crash and restart streams deliberately.
Keep binlog retention long enough for stream catchup windows.
Run vreplication health checks to detect stuck streams early.

When it happens

Trigger: Any error during BinlogStreamer.Stream besides ErrBinlogUnavailable: NewBinlogConnection failure, dump-command failure, event read/parse errors, or transaction-sequence mismatches. The wrapper preserves the wrapped error and annotates it with the position at failure.

Common situations: Primary failover/crash mid-stream; network drops during long-running reshard or filtered-replication streams; corrupted binlog events; GTID sequence gaps after a partial restore causing a non-unavailable stream failure.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/0dfddceab5e3a886. Report an issue: GitHub.

Appendix: source

Thrown at go/vt/binlog/binlog_streamer.go:184

		se:              se,
		clientCharset:   clientCharset,
		startPos:        startPos,
		timestamp:       timestamp,
		sendTransaction: sendTransaction,
	}
}

// Stream starts streaming binlog events using the settings from NewStreamer().
func (bls *Streamer) Stream(ctx context.Context) (err error) {
	// Ensure se is Open. If vttablet came up in a non_serving role,
	// the schema engine may not have been initialized.
	if err := bls.se.Open(); err != nil {
		return err
	}
	stopPos := bls.startPos
	defer func() {
		if err != nil && err != ErrBinlogUnavailable {
			err = fmt.Errorf("stream error @ (including the GTID we failed to process) %v: %v", stopPos, err)
		}
		log.Info(fmt.Sprintf("stream ended @ %v, err = %v", stopPos, err))
	}()

	if bls.conn, err = NewBinlogConnection(bls.cp); err != nil {
		return err
	}
	defer bls.conn.Close()

	// Check that the default charsets match, if the client specified one.
	// Note that Streamer uses the settings for the 'dba' user, while
	// BinlogPlayer uses the 'filtered' user, so those are the ones whose charset
	// must match. Filtered replication should still succeed even with a default
	// mismatch, since we pass per-statement charset info. However, Vitess in
	// general doesn't support servers with different default charsets, so we
	// treat it as a configuration error.
	if bls.clientCharset != nil {
		cs, err := mysql.GetCharset(bls.conn.Conn)

View on GitHub (pinned to 01a25a7d17)