{"record":{"id":"8dfbf227030ba562","repo":"vitessio/vitess","slug":"panic-in-s-v","errorCode":null,"errorMessage":"panic in %s: %v","messagePattern":"panic in (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vttablet/tabletmanager/vreplication/vplayer.go","lineNumber":266,"sourceCode":"\t}\n\treturn nil\n}\n\n// runWithRecover invokes fn and converts any panic into an error so the\n// caller's child goroutine doesn't crash the entire vttablet process. The\n// panic value and stack are logged with workflow context. Used to wrap\n// fetchAndApply's child goroutine bodies (#20360) which would otherwise\n// escape controller.runBlp's recover (that runs on a different goroutine).\nfunc runWithRecover(workflow, where string, fn func() error) (err error) {\n\tdefer func() {\n\t\tif x := recover(); x != nil {\n\t\t\tlog.Error(\"caught panic\",\n\t\t\t\tslog.String(\"workflow\", workflow),\n\t\t\t\tslog.String(\"where\", where),\n\t\t\t\tslog.Any(\"panic\", x),\n\t\t\t\tslog.String(\"stack\", string(tb.Stack(4))),\n\t\t\t)\n\t\t\terr = fmt.Errorf(\"panic in %s: %v\", where, x)\n\t\t}\n\t}()\n\treturn fn()\n}\n\n// fetchAndApply performs the fetching and application of the binlogs.\n// This is done by two different threads. The fetcher thread pulls\n// events from the vstreamer and adds them to the relayLog.\n// The applyEvents thread pulls accumulated events from the relayLog\n// to apply them to mysql. The reason for this separation is because\n// commits are slow during apply. So, more events can accumulate in\n// the relay log during a commit. In such situations, the next iteration\n// of apply combines all the transactions in the relay log into a single\n// one. This allows for the apply thread to catch up more quickly if\n// a backlog builds up.\nfunc (vp *vplayer) fetchAndApply(ctx context.Context) (err error) {\n\tlog.Info(fmt.Sprintf(\"Starting VReplication player id: %v, name: %v, startPos: %v, stop: %v\", vp.vr.id, vp.vr.WorkflowName, vp.startPos, vp.stopPos))\n","sourceCodeStart":248,"sourceCodeEnd":284,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vttablet/tabletmanager/vreplication/vplayer.go#L248-L284","documentation":"This error is produced by vplayer's deferred recover() handler when the wrapped function (fetchAndApply or similar) panics. The recover converts the panic into a Go error so vreplication can record it and stop/retry gracefully instead of crashing the tablet process. The %v is the function/location name, and %v is the panic value.","triggerScenarios":"Any panic inside the wrapped vreplication execution path: nil pointer dereference on event data, index out of range while parsing row events, panics in third-party/plan-building code, etc. The deferred func catches x and formats it into this error.","commonSituations":"Unexpected binlog event shapes from unusual source DDL or MySQL versions; bugs in table plan construction; concurrent modification of shared state; OOM-adjacent conditions manifesting as nil dereferences.","solutions":["Read the full stack trace logged alongside ('caught panic' log with slog stack field) to locate the panicking code","Check the tablet logs for the workflow/table involved and reproduce with that specific event or schema","Upgrade Vitess — many vplayer panics are fixed in later releases; check release notes","If caused by anomalous source data/DDL, normalize the source schema and restart the workflow"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// vplayer already recovers the panic; on the caller side, handle the returned error\nif err := vp.fetchAndApply(ctx, events); err != nil {\n    if strings.HasPrefix(err.Error(), \"panic in \") {\n        // capture tablet logs + stack, stop the workflow, report to Vitess\n        log.Error(\"vreplication panic converted to error\", slog.Any(\"error\", err))\n    }\n}","preventionTips":["Keep Vitess up to date — many panic paths are fixed in later releases","Avoid unvetted custom patches to the vreplication event loop","Ensure source binlog events match expected shapes (ROW format, supported MySQL version)","Monitor tablet logs for 'caught panic' and triage stacks immediately"],"tags":["vreplication","panic","recovery","vplayer"],"backgroundTag":"panic-recovered-as-error","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}