vitessio/vitess · error

panic in %s: %v

Error message

panic in %s: %v

What it means

This error is produced by vplayer's deferred recover() handler when the wrapped function (fetchAndApply or similar) panics. The recover converts the panic into a Go error so vreplication can record it and stop/retry gracefully instead of crashing the tablet process. The %v is the function/location name, and %v is the panic value.

Source

Thrown at go/vt/vttablet/tabletmanager/vreplication/vplayer.go:266

	}
	return nil
}

// runWithRecover invokes fn and converts any panic into an error so the
// caller's child goroutine doesn't crash the entire vttablet process. The
// panic value and stack are logged with workflow context. Used to wrap
// fetchAndApply's child goroutine bodies (#20360) which would otherwise
// escape controller.runBlp's recover (that runs on a different goroutine).
func runWithRecover(workflow, where string, fn func() error) (err error) {
	defer func() {
		if x := recover(); x != nil {
			log.Error("caught panic",
				slog.String("workflow", workflow),
				slog.String("where", where),
				slog.Any("panic", x),
				slog.String("stack", string(tb.Stack(4))),
			)
			err = fmt.Errorf("panic in %s: %v", where, x)
		}
	}()
	return fn()
}

// fetchAndApply performs the fetching and application of the binlogs.
// This is done by two different threads. The fetcher thread pulls
// events from the vstreamer and adds them to the relayLog.
// The applyEvents thread pulls accumulated events from the relayLog
// to apply them to mysql. The reason for this separation is because
// commits are slow during apply. So, more events can accumulate in
// the relay log during a commit. In such situations, the next iteration
// of apply combines all the transactions in the relay log into a single
// one. This allows for the apply thread to catch up more quickly if
// a backlog builds up.
func (vp *vplayer) fetchAndApply(ctx context.Context) (err error) {
	log.Info(fmt.Sprintf("Starting VReplication player id: %v, name: %v, startPos: %v, stop: %v", vp.vr.id, vp.vr.WorkflowName, vp.startPos, vp.stopPos))

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Read the full stack trace logged alongside ('caught panic' log with slog stack field) to locate the panicking code
2. Check the tablet logs for the workflow/table involved and reproduce with that specific event or schema
3. Upgrade Vitess — many vplayer panics are fixed in later releases; check release notes
4. If caused by anomalous source data/DDL, normalize the source schema and restart the workflow

// vplayer already recovers the panic; on the caller side, handle the returned error
if err := vp.fetchAndApply(ctx, events); err != nil {
    if strings.HasPrefix(err.Error(), "panic in ") {
        // capture tablet logs + stack, stop the workflow, report to Vitess
        log.Error("vreplication panic converted to error", slog.Any("error", err))
    }
}

When it happens

Trigger: Any panic inside the wrapped vreplication execution path: nil pointer dereference on event data, index out of range while parsing row events, panics in third-party/plan-building code, etc. The deferred func catches x and formats it into this error.

Common situations: Unexpected binlog event shapes from unusual source DDL or MySQL versions; bugs in table plan construction; concurrent modification of shared state; OOM-adjacent conditions manifesting as nil dereferences.

Related errors

• progress stalled; vplayer was unable to replicate the transa
• vstream ended
• uncaught panic: %v
• uncaught panic: %v, vtgate: %v
• caught panic during %v: %v

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/8dfbf227030ba562. Report an issue: GitHub.