vitessio/vitess · critical
stream error @ (including the GTID we failed to process) %v:
Error message
stream error @ (including the GTID we failed to process) %v: %v
What it means
In binlog_streamer.go, when streaming ends with an error other than ErrBinlogUnavailable, the deferred handler wraps the error with the last known position (stopPos) as "stream error @ (including the GTID we failed to process) <position>: <original>". The position marker tells operators exactly which GTID/binlog position the streamer had reached (and could not get past) when the failure occurred, since the stop position is updated as events are successfully processed.
Source
Thrown at go/vt/binlog/binlog_streamer.go:184
se: se,
clientCharset: clientCharset,
startPos: startPos,
timestamp: timestamp,
sendTransaction: sendTransaction,
}
}
// Stream starts streaming binlog events using the settings from NewStreamer().
func (bls *Streamer) Stream(ctx context.Context) (err error) {
// Ensure se is Open. If vttablet came up in a non_serving role,
// the schema engine may not have been initialized.
if err := bls.se.Open(); err != nil {
return err
}
stopPos := bls.startPos
defer func() {
if err != nil && err != ErrBinlogUnavailable {
err = fmt.Errorf("stream error @ (including the GTID we failed to process) %v: %v", stopPos, err)
}
log.Info(fmt.Sprintf("stream ended @ %v, err = %v", stopPos, err))
}()
if bls.conn, err = NewBinlogConnection(bls.cp); err != nil {
return err
}
defer bls.conn.Close()
// Check that the default charsets match, if the client specified one.
// Note that Streamer uses the settings for the 'dba' user, while
// BinlogPlayer uses the 'filtered' user, so those are the ones whose charset
// must match. Filtered replication should still succeed even with a default
// mismatch, since we pass per-statement charset info. However, Vitess in
// general doesn't support servers with different default charsets, so we
// treat it as a configuration error.
if bls.clientCharset != nil {
cs, err := mysql.GetCharset(bls.conn.Conn)View on GitHub (pinned to 01a25a7d17)
Solutions
- Unwrap the error: the %v suffix carries the root cause; fix that first (reconnect, corruption, permissions, etc.).
- Use the @ position in the message to resume streaming from (or inspect the binlog at) the exact GTID that failed.
- If the primary failed over, reparent and restart the stream — the streamer will reconnect from the recorded position.
- If the failing GTID corresponds to a corrupt/unavailable binlog file, restore it from backup or use a vreplication copy/reshard to re-sync.
- Check MySQL error logs and binlog integrity (mysqlbinlog) at the reported position.
Example fix
// before: treats all stream failures identically
if err != nil {
return err
}
// after: distinguish expected unavailability from real stream failures
if errors.Is(err, ErrBinlogUnavailable) {
// wait/refresh binlog sources, then retry
} else if err != nil {
log.Error("binlog stream failed",
slog.Any("position", stopPos), slog.Any("error", err))
return err
} Defensive patterns
Strategy: try-catch
Validate before calling
// before streaming, confirm the starting position exists in the primary's binlogs
qr, _ := conn.ExecuteFetch("SHOW BINARY LOG STATUS", 1, false)
_ = qr // compare startPos against current file list/GTID set Try / catch
err := bls.Stream(ctx)
if err != nil {
var stopPos replication.Position
if m := regexp.MustCompile(`stream error @ (\([^)]*\))?[^:]*: `).FindStringSubmatch(err.Error()); m != nil {
_ = m // resume from this position after fixing root cause
}
if strings.Contains(err.Error(), "ErrBinlogUnavailable") {
// wait for binlogs / refresh sources
}
} Prevention
- Always log and persist the stream position on failure so restarts resume at the right GTID.
- Distinguish ErrBinlogUnavailable from other failures before retrying.
- Monitor primaries for failover/crash and restart streams deliberately.
- Keep binlog retention long enough for stream catchup windows.
- Run vreplication health checks to detect stuck streams early.
When it happens
Trigger: Any error during BinlogStreamer.Stream besides ErrBinlogUnavailable: NewBinlogConnection failure, dump-command failure, event read/parse errors, or transaction-sequence mismatches. The wrapper preserves the wrapped error and annotates it with the position at failure.
Common situations: Primary failover/crash mid-stream; network drops during long-running reshard or filtered-replication streams; corrupted binlog events; GTID sequence gaps after a partial restore causing a non-unavailable stream failure.
Related errors
- GetPreviousGTIDs: previous GTIDs not found
- startPos.GTIDSet is wrong type - expected filePosGTID, got:
- failed to get primary position: %v
- failed to set @source_binlog_checksum=@@global.binlog_checks
- failed to send the ComBinlogDump command: %v
AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01).
Data as JSON: /api/errors/0dfddceab5e3a886.
Report an issue: GitHub.