{"record":{"id":"74036229bc3f6a5c","repo":"vitessio/vitess","slug":"panic-v","errorCode":null,"errorMessage":"panic: %v","messagePattern":"panic: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"go/vt/vttablet/tabletmanager/vreplication/controller.go","lineNumber":284,"sourceCode":"\tif _, err := dbClient.ExecuteFetch(fmt.Sprintf(\"set @@session.net_write_timeout = %v\",\n\t\tworkflowConfig.NetWriteTimeout), maxRows); err != nil {\n\t\treturn err\n\t}\n\t// We must apply AUTO_INCREMENT values precisely as we got them. This include the 0 value, which is\n\t// not recommended in AUTO_INCREMENT, and yet is valid.\n\tif _, err := dbClient.ExecuteFetch(\"set @@session.sql_mode = CONCAT(@@session.sql_mode, ',NO_AUTO_VALUE_ON_ZERO')\",\n\t\tmaxRows); err != nil {\n\t\treturn err\n\t}\n\treturn nil\n}\n\nfunc (ct *controller) runBlp(ctx context.Context) (err error) {\n\tdefer func() {\n\t\tct.sourceTablet.Store(&topodatapb.TabletAlias{})\n\t\tif x := recover(); x != nil {\n\t\t\tlog.Error(fmt.Sprintf(\"%s caught panic: %v\\n%s\", ct.logPrefix(), x, tb.Stack(4)))\n\t\t\terr = fmt.Errorf(\"panic: %v\", x)\n\t\t}\n\t}()\n\n\tselect {\n\tcase <-ctx.Done():\n\t\treturn nil\n\tdefault:\n\t}\n\n\tdbClient := ct.dbClientFactory()\n\tif err := dbClient.Connect(); err != nil {\n\t\treturn vterrors.Wrap(err, \"can't connect to database\")\n\t}\n\tdefer dbClient.Close()\n\n\ttablet, err := ct.pickSourceTablet(ctx, dbClient)\n\tif err != nil {\n\t\treturn err","sourceCodeStart":266,"sourceCodeEnd":302,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vttablet/tabletmanager/vreplication/controller.go#L266-L302","documentation":"The vreplication controller's runBlp goroutine recovered from a panic (nil map access, nil pointer dereference, etc. anywhere inside the binlog player loop) and converts it into an error via `fmt.Errorf(\"panic: %v\", x)`. The controller stores an empty source tablet alias and returns the panic text as the run error so vreplication state machinery can record the failure.","triggerScenarios":"Any panic inside the vreplication controller's copy loop (runBlp) — e.g. nil dereference while processing binlog events, index out of range in the player, or a panicking tablet picker — surfaced when the deferred recover() fires.","commonSituations":"Bug in Vitess itself or a plugin/hook triggered by unusual binlog content; resource exhaustion corrupting state; version mismatch between vttablet and MySQL causing unexpected wire data.","solutions":["Check the accompanying 'caught panic: ...' log line and the Go stack trace (tb.Stack) to identify the panicking function.","Restart the tablet so vreplication can resume from the last committed position.","Search the panic stack against the Vitess issue tracker; upgrade to a release containing the fix.","If reproducible, capture the binlog position and file a bug with the stack trace and stream id."],"exampleFix":"// The error is produced internally by the recover:\n// before (panic propagates, goroutine dies without state update)\n// after (Vitess controller code)\ndefer func() {\n\tif x := recover(); x != nil {\n\t\tlog.Error(fmt.Sprintf(\"%s caught panic: %v\\n%s\", ct.logPrefix(), x, tb.Stack(4)))\n\t\terr = fmt.Errorf(\"panic: %v\", x)\n\t}\n}()","handlingStrategy":"retry","validationCode":"// No caller-side pre-check; ensure topology & MySQL reachable so the player starts cleanly:\nvtctldclient GetTablets --keyspace <source> --shard <shard>","typeGuard":null,"tryCatchPattern":"// Treat the returned 'panic: ...' error as a retryable crash:\nfor attempt := 0; attempt < 3; attempt++ {\n    err := controller.runBlp(ctx)\n    if err == nil { break }\n    if strings.HasPrefix(err.Error(), \"panic:\") {\n        log.Warn(\"vreplication controller panicked, retrying\", slog.Any(\"error\", err))\n        time.Sleep(backoff(attempt))\n        continue\n    }\n    return err\n}","preventionTips":["Keep vttablet upgraded to the latest patch release to pick up panic fixes in the binlog player.","Watch logs for the 'caught panic' message with stack traces and report reproducible cases upstream.","Avoid unsupported MySQL flavors/versions that may emit unexpected binlog events."],"tags":["go","panic","vreplication","recovered-panic"],"backgroundTag":"goroutine-panic-recovered","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}