{"record":{"id":"1cf06d25344a45a0","repo":"vitessio/vitess","slug":"caught-panic-during-v-v","errorCode":null,"errorMessage":"caught panic during %v: %v","messagePattern":"caught panic during (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vttablet/tabletmanager/rpc_server.go","lineNumber":53,"sourceCode":"//\n\n// lock is used at the beginning of an RPC call, to acquire the\n// action semaphore. It returns ctx.Err() if the context expires.\nfunc (tm *TabletManager) lock(ctx context.Context) error {\n\treturn tm.actionSema.Acquire(ctx, 1)\n}\n\n// unlock is the symmetrical action to lock.\nfunc (tm *TabletManager) unlock() {\n\ttm.actionSema.Release(1)\n}\n\n// HandleRPCPanic is part of the RPCTM interface.\nfunc (tm *TabletManager) HandleRPCPanic(ctx context.Context, name string, args, reply any, verbose bool, err *error) {\n\t// panic handling\n\tif x := recover(); x != nil {\n\t\tlog.Error(fmt.Sprintf(\"TabletManager.%v(%v) on %v panic: %v\\n%s\", name, args, topoproto.TabletAliasString(tm.tabletAlias), x, tb.Stack(4)))\n\t\t*err = fmt.Errorf(\"caught panic during %v: %v\", name, x)\n\t\treturn\n\t}\n\n\t// quick check for fast path\n\tif !verbose && *err == nil {\n\t\treturn\n\t}\n\n\t// we gotta log something, get the source\n\tfrom := \"\"\n\tci, ok := callinfo.FromContext(ctx)\n\tif ok {\n\t\tfrom = ci.Text()\n\t}\n\n\tif *err != nil {\n\t\t// error case\n\t\trootCause := vterrors.RootCause(*err)","sourceCodeStart":35,"sourceCodeEnd":71,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vttablet/tabletmanager/rpc_server.go#L35-L71","documentation":"HandleRPCPanic is the deferred panic recovery wrapper for TabletManager RPCs. When a managed RPC panics, the panic value is converted into this error ('caught panic during <rpc-name>: <panic>') and returned to the RPC caller instead of crashing the vttablet process.","triggerScenarios":"Any TabletManager RPC that panics internally — e.g. nil pointer dereferences, index-out-of-range, or third-party library panics during backup, snapshot, or tablet type change calls.","commonSituations":"Bugs in vttablet code paths or plugins triggered by unusual tablet states, corrupted local data causing panics during backup/restore, version mismatch between vtctld and vttablet exposing unhandled paths.","solutions":["Read the full panic stack trace in the vttablet log line immediately above the error","Report/reproduce the panic as a vitess bug if it appears to be a code defect","Check for data or environment issues (disk full, corrupt backup) that the RPC was handling","Retry the RPC after fixing the underlying condition; the process itself was not crashed"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := tm RPC call (e.g. Backup, ChangeType)\nif err != nil && strings.HasPrefix(err.Error(), \"caught panic during \") {\n    rpcName := strings.TrimPrefix(err.Error(), \"caught panic during \")\n    log.Errorf(\"tabletmanager RPC %s panicked; check vttablet logs for stack trace\", rpcName)\n    // do not blind-retry: inspect the vttablet stack trace first\n}","preventionTips":["Scan vttablet logs for 'panic:' lines to catch latent bugs early","Keep vttablet and vtctld on matching versions","Validate inputs to tablet RPCs (aliases, paths) before calls","Report reproducible panics upstream with the full stack trace"],"tags":["panic","tabletmanager","rpc","recovery"],"backgroundTag":"rpc-panic-recovered","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}