vitessio/vitess · error

caught panic during %v: %v

Error message

caught panic during %v: %v

What it means

HandleRPCPanic is the deferred panic recovery wrapper for TabletManager RPCs. When a managed RPC panics, the panic value is converted into this error ('caught panic during <rpc-name>: <panic>') and returned to the RPC caller instead of crashing the vttablet process.

Source

Thrown at go/vt/vttablet/tabletmanager/rpc_server.go:53

//

// lock is used at the beginning of an RPC call, to acquire the
// action semaphore. It returns ctx.Err() if the context expires.
func (tm *TabletManager) lock(ctx context.Context) error {
	return tm.actionSema.Acquire(ctx, 1)
}

// unlock is the symmetrical action to lock.
func (tm *TabletManager) unlock() {
	tm.actionSema.Release(1)
}

// HandleRPCPanic is part of the RPCTM interface.
func (tm *TabletManager) HandleRPCPanic(ctx context.Context, name string, args, reply any, verbose bool, err *error) {
	// panic handling
	if x := recover(); x != nil {
		log.Error(fmt.Sprintf("TabletManager.%v(%v) on %v panic: %v\n%s", name, args, topoproto.TabletAliasString(tm.tabletAlias), x, tb.Stack(4)))
		*err = fmt.Errorf("caught panic during %v: %v", name, x)
		return
	}

	// quick check for fast path
	if !verbose && *err == nil {
		return
	}

	// we gotta log something, get the source
	from := ""
	ci, ok := callinfo.FromContext(ctx)
	if ok {
		from = ci.Text()
	}

	if *err != nil {
		// error case
		rootCause := vterrors.RootCause(*err)

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Read the full panic stack trace in the vttablet log line immediately above the error
2. Report/reproduce the panic as a vitess bug if it appears to be a code defect
3. Check for data or environment issues (disk full, corrupt backup) that the RPC was handling
4. Retry the RPC after fixing the underlying condition; the process itself was not crashed

err := tm RPC call (e.g. Backup, ChangeType)
if err != nil && strings.HasPrefix(err.Error(), "caught panic during ") {
    rpcName := strings.TrimPrefix(err.Error(), "caught panic during ")
    log.Errorf("tabletmanager RPC %s panicked; check vttablet logs for stack trace", rpcName)
    // do not blind-retry: inspect the vttablet stack trace first
}

When it happens

Trigger: Any TabletManager RPC that panics internally — e.g. nil pointer dereferences, index-out-of-range, or third-party library panics during backup, snapshot, or tablet type change calls.

Common situations: Bugs in vttablet code paths or plugins triggered by unusual tablet states, corrupted local data causing panics during backup/restore, version mismatch between vtctld and vttablet exposing unhandled paths.

Related errors

• uncaught panic: %v
• uncaught panic: %v, vtgate: %v
• panic in %s: %v
• invalid key:value pair
• this command panics on purpose

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/1cf06d25344a45a0. Report an issue: GitHub.