vitessio/vitess · error

progress stalled; vplayer was unable to replicate the transa

Error message

progress stalled; vplayer was unable to replicate the transaction in a timely manner; examine the target mysqld instance health and the replicated queries' EXPLAIN output to see why queries are taking unusually long

What it means

errVPlayerStalled is returned when the vplayer (binlog player) has made no replication progress for vplayerProgressDeadline (5 minutes). It signals that transactions from the source are being applied but take abnormally long, so vreplication is effectively stalled.

Source

Thrown at go/vt/vttablet/tabletmanager/vreplication/vplayer.go:53

	"vitess.io/vitess/go/vt/log"
	"vitess.io/vitess/go/vt/vterrors"
	vttablet "vitess.io/vitess/go/vt/vttablet/common"
	"vitess.io/vitess/go/vt/vttablet/tabletserver/throttle/throttlerapp"

	binlogdatapb "vitess.io/vitess/go/vt/proto/binlogdata"
	vtrpcpb "vitess.io/vitess/go/vt/proto/vtrpc"
)

const failedToRecordHeartbeatMsg = "failed to record heartbeat"

var (
	// At what point should we consider the vplayer to be stalled and return an error.
	// 5 minutes is well beyond a reasonable amount of time for a transaction to be
	// replicated.
	vplayerProgressDeadline = time.Duration(5 * time.Minute)

	// The error to return when we have detected a stall in the vplayer.
	errVPlayerStalled = errors.New("progress stalled; vplayer was unable to replicate the transaction in a timely manner; examine the target mysqld instance health and the replicated queries' EXPLAIN output to see why queries are taking unusually long")

	// vplayerThrottleEpoch is the reference instant for throttle-denial
	// offsets: it carries a monotonic clock reading, so durations measured
	// against it are immune to wall clock steps in either direction.
	vplayerThrottleEpoch = time.Now()
)

// vplayer replays binlog events by pulling them from a vstreamer.
type vplayer struct {
	vr        *vreplicator
	startPos  replication.Position
	stopPos   replication.Position
	saveStop  bool
	copyState map[string]*sqltypes.Result

	replicatorPlan *ReplicatorPlan
	tablePlans     map[string]*TablePlan

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Run EXPLAIN on the queries being replicated (check vreplication logs for the stalled transaction) and add missing indexes/primary keys on the target tables
2. Check target mysqld health: slow query log, locks (SHOW PROCESSLIST / INNODB STATUS), disk I/O, and replication lag
3. Reduce transaction size on the source (break up huge batch DMLs) and verify throttler settings are not starving the vreplication stream

Example fix

// before (target table missing index, DML full-scans)
CREATE TABLE t (id INT, name VARCHAR(64));
// after
CREATE TABLE t (id INT PRIMARY KEY, name VARCHAR(64), KEY idx_name (name));

// Before/while vreplication runs, check target health and slow queries:
// SHOW PROCESSLIST — long-running queries on vreplication tables
// SELECT * FROM information_schema.innodb_trx WHERE trx_started < NOW() - INTERVAL 5 MINUTE

if err := stream.Send(ctx); err != nil {
    if strings.Contains(err.Error(), "progress stalled") {
        log.Warn("vreplication stalled; checking target health", slog.Any("error", err))
        // alert + inspect EXPLAIN of the stalled txn, then let vreplication retry
    }
    return err
}

When it happens

Trigger: Send/recordHeartbeat/TestRelayLogSendStallDeferredWhileThrottled observe no heartbeat/progress advance within 5 minutes while applying transactions on the target mysqld.

Common situations: Long-running or unindexed DMLs on the target (missing primary/secondary keys causing full table scans); target mysqld overloaded, locked, or disk-starved; very large single transactions from the source; throttling combined with slow apply.

Related errors

• vstream ended
• panic in %s: %v
• internal error: vplayer is in a transaction on event: %v
• value out of range
• both atomic copy and partial mode cannot be specified for th

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/19d8fa965f7b897e. Report an issue: GitHub.