vitessio/vitess · critical

panic: %v

Error message

panic: %v

What it means

The vreplication controller's runBlp goroutine recovered from a panic (nil map access, nil pointer dereference, etc. anywhere inside the binlog player loop) and converts it into an error via `fmt.Errorf("panic: %v", x)`. The controller stores an empty source tablet alias and returns the panic text as the run error so vreplication state machinery can record the failure.

Source

Thrown at go/vt/vttablet/tabletmanager/vreplication/controller.go:284

	if _, err := dbClient.ExecuteFetch(fmt.Sprintf("set @@session.net_write_timeout = %v",
		workflowConfig.NetWriteTimeout), maxRows); err != nil {
		return err
	}
	// We must apply AUTO_INCREMENT values precisely as we got them. This include the 0 value, which is
	// not recommended in AUTO_INCREMENT, and yet is valid.
	if _, err := dbClient.ExecuteFetch("set @@session.sql_mode = CONCAT(@@session.sql_mode, ',NO_AUTO_VALUE_ON_ZERO')",
		maxRows); err != nil {
		return err
	}
	return nil
}

func (ct *controller) runBlp(ctx context.Context) (err error) {
	defer func() {
		ct.sourceTablet.Store(&topodatapb.TabletAlias{})
		if x := recover(); x != nil {
			log.Error(fmt.Sprintf("%s caught panic: %v\n%s", ct.logPrefix(), x, tb.Stack(4)))
			err = fmt.Errorf("panic: %v", x)
		}
	}()

	select {
	case <-ctx.Done():
		return nil
	default:
	}

	dbClient := ct.dbClientFactory()
	if err := dbClient.Connect(); err != nil {
		return vterrors.Wrap(err, "can't connect to database")
	}
	defer dbClient.Close()

	tablet, err := ct.pickSourceTablet(ctx, dbClient)
	if err != nil {
		return err

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check the accompanying 'caught panic: ...' log line and the Go stack trace (tb.Stack) to identify the panicking function.
2. Restart the tablet so vreplication can resume from the last committed position.
3. Search the panic stack against the Vitess issue tracker; upgrade to a release containing the fix.
4. If reproducible, capture the binlog position and file a bug with the stack trace and stream id.

Example fix

// The error is produced internally by the recover:
// before (panic propagates, goroutine dies without state update)
// after (Vitess controller code)
defer func() {
	if x := recover(); x != nil {
		log.Error(fmt.Sprintf("%s caught panic: %v\n%s", ct.logPrefix(), x, tb.Stack(4)))
		err = fmt.Errorf("panic: %v", x)
	}
}()

// No caller-side pre-check; ensure topology & MySQL reachable so the player starts cleanly:
vtctldclient GetTablets --keyspace <source> --shard <shard>

// Treat the returned 'panic: ...' error as a retryable crash:
for attempt := 0; attempt < 3; attempt++ {
    err := controller.runBlp(ctx)
    if err == nil { break }
    if strings.HasPrefix(err.Error(), "panic:") {
        log.Warn("vreplication controller panicked, retrying", slog.Any("error", err))
        time.Sleep(backoff(attempt))
        continue
    }
    return err
}

When it happens

Trigger: Any panic inside the vreplication controller's copy loop (runBlp) — e.g. nil dereference while processing binlog events, index out of range in the player, or a panicking tablet picker — surfaced when the deferred recover() fires.

Common situations: Bug in Vitess itself or a plugin/hook triggered by unusual binlog content; resource exhaustion corrupting state; version mismatch between vttablet and MySQL causing unexpected wire data.

Related errors

• QueryRowsMap unexpected error: %+v
• ExecNoPrepare unexpected error: %+v
• %w in keyspace %s for %s
• vreplication streams are not frozen on tablet %d
• routing still exists from keyspace %s table %s to %s

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/74036229bc3f6a5c. Report an issue: GitHub.