{"record":{"id":"0dfddceab5e3a886","repo":"vitessio/vitess","slug":"stream-error-including-the-gtid-we-failed-to-pr","errorCode":null,"errorMessage":"stream error @ (including the GTID we failed to process) %v: %v","messagePattern":"stream error @ \\(including the GTID we failed to process\\) (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"go/vt/binlog/binlog_streamer.go","lineNumber":184,"sourceCode":"\t\tse:              se,\n\t\tclientCharset:   clientCharset,\n\t\tstartPos:        startPos,\n\t\ttimestamp:       timestamp,\n\t\tsendTransaction: sendTransaction,\n\t}\n}\n\n// Stream starts streaming binlog events using the settings from NewStreamer().\nfunc (bls *Streamer) Stream(ctx context.Context) (err error) {\n\t// Ensure se is Open. If vttablet came up in a non_serving role,\n\t// the schema engine may not have been initialized.\n\tif err := bls.se.Open(); err != nil {\n\t\treturn err\n\t}\n\tstopPos := bls.startPos\n\tdefer func() {\n\t\tif err != nil && err != ErrBinlogUnavailable {\n\t\t\terr = fmt.Errorf(\"stream error @ (including the GTID we failed to process) %v: %v\", stopPos, err)\n\t\t}\n\t\tlog.Info(fmt.Sprintf(\"stream ended @ %v, err = %v\", stopPos, err))\n\t}()\n\n\tif bls.conn, err = NewBinlogConnection(bls.cp); err != nil {\n\t\treturn err\n\t}\n\tdefer bls.conn.Close()\n\n\t// Check that the default charsets match, if the client specified one.\n\t// Note that Streamer uses the settings for the 'dba' user, while\n\t// BinlogPlayer uses the 'filtered' user, so those are the ones whose charset\n\t// must match. Filtered replication should still succeed even with a default\n\t// mismatch, since we pass per-statement charset info. However, Vitess in\n\t// general doesn't support servers with different default charsets, so we\n\t// treat it as a configuration error.\n\tif bls.clientCharset != nil {\n\t\tcs, err := mysql.GetCharset(bls.conn.Conn)","sourceCodeStart":166,"sourceCodeEnd":202,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/binlog/binlog_streamer.go#L166-L202","documentation":"In binlog_streamer.go, when streaming ends with an error other than ErrBinlogUnavailable, the deferred handler wraps the error with the last known position (stopPos) as \"stream error @ (including the GTID we failed to process) <position>: <original>\". The position marker tells operators exactly which GTID/binlog position the streamer had reached (and could not get past) when the failure occurred, since the stop position is updated as events are successfully processed.","triggerScenarios":"Any error during BinlogStreamer.Stream besides ErrBinlogUnavailable: NewBinlogConnection failure, dump-command failure, event read/parse errors, or transaction-sequence mismatches. The wrapper preserves the wrapped error and annotates it with the position at failure.","commonSituations":"Primary failover/crash mid-stream; network drops during long-running reshard or filtered-replication streams; corrupted binlog events; GTID sequence gaps after a partial restore causing a non-unavailable stream failure.","solutions":["Unwrap the error: the %v suffix carries the root cause; fix that first (reconnect, corruption, permissions, etc.).","Use the @ position in the message to resume streaming from (or inspect the binlog at) the exact GTID that failed.","If the primary failed over, reparent and restart the stream — the streamer will reconnect from the recorded position.","If the failing GTID corresponds to a corrupt/unavailable binlog file, restore it from backup or use a vreplication copy/reshard to re-sync.","Check MySQL error logs and binlog integrity (mysqlbinlog) at the reported position."],"exampleFix":"// before: treats all stream failures identically\nif err != nil {\n\treturn err\n}\n// after: distinguish expected unavailability from real stream failures\nif errors.Is(err, ErrBinlogUnavailable) {\n\t// wait/refresh binlog sources, then retry\n} else if err != nil {\n\tlog.Error(\"binlog stream failed\",\n\t\tslog.Any(\"position\", stopPos), slog.Any(\"error\", err))\n\treturn err\n}","handlingStrategy":"try-catch","validationCode":"// before streaming, confirm the starting position exists in the primary's binlogs\nqr, _ := conn.ExecuteFetch(\"SHOW BINARY LOG STATUS\", 1, false)\n_ = qr // compare startPos against current file list/GTID set","typeGuard":null,"tryCatchPattern":"err := bls.Stream(ctx)\nif err != nil {\n\tvar stopPos replication.Position\n\tif m := regexp.MustCompile(`stream error @ (\\([^)]*\\))?[^:]*: `).FindStringSubmatch(err.Error()); m != nil {\n\t\t_ = m // resume from this position after fixing root cause\n\t}\n\tif strings.Contains(err.Error(), \"ErrBinlogUnavailable\") {\n\t\t// wait for binlogs / refresh sources\n\t}\n}","preventionTips":["Always log and persist the stream position on failure so restarts resume at the right GTID.","Distinguish ErrBinlogUnavailable from other failures before retrying.","Monitor primaries for failover/crash and restart streams deliberately.","Keep binlog retention long enough for stream catchup windows.","Run vreplication health checks to detect stuck streams early."],"tags":["mysql","replication","binlog","streaming","gtid"],"backgroundTag":"binlog-stream-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}