{"record":{"id":"19d8fa965f7b897e","repo":"vitessio/vitess","slug":"progress-stalled-vplayer-was-unable-to-replicate","errorCode":null,"errorMessage":"progress stalled; vplayer was unable to replicate the transaction in a timely manner; examine the target mysqld instance health and the replicated queries' EXPLAIN output to see why queries are taking unusually long","messagePattern":"progress stalled; vplayer was unable to replicate the transaction in a timely manner; examine the target mysqld instance health and the replicated queries' EXPLAIN output to see why queries are taking unusually long","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vttablet/tabletmanager/vreplication/vplayer.go","lineNumber":53,"sourceCode":"\t\"vitess.io/vitess/go/vt/log\"\n\t\"vitess.io/vitess/go/vt/vterrors\"\n\tvttablet \"vitess.io/vitess/go/vt/vttablet/common\"\n\t\"vitess.io/vitess/go/vt/vttablet/tabletserver/throttle/throttlerapp\"\n\n\tbinlogdatapb \"vitess.io/vitess/go/vt/proto/binlogdata\"\n\tvtrpcpb \"vitess.io/vitess/go/vt/proto/vtrpc\"\n)\n\nconst failedToRecordHeartbeatMsg = \"failed to record heartbeat\"\n\nvar (\n\t// At what point should we consider the vplayer to be stalled and return an error.\n\t// 5 minutes is well beyond a reasonable amount of time for a transaction to be\n\t// replicated.\n\tvplayerProgressDeadline = time.Duration(5 * time.Minute)\n\n\t// The error to return when we have detected a stall in the vplayer.\n\terrVPlayerStalled = errors.New(\"progress stalled; vplayer was unable to replicate the transaction in a timely manner; examine the target mysqld instance health and the replicated queries' EXPLAIN output to see why queries are taking unusually long\")\n\n\t// vplayerThrottleEpoch is the reference instant for throttle-denial\n\t// offsets: it carries a monotonic clock reading, so durations measured\n\t// against it are immune to wall clock steps in either direction.\n\tvplayerThrottleEpoch = time.Now()\n)\n\n// vplayer replays binlog events by pulling them from a vstreamer.\ntype vplayer struct {\n\tvr        *vreplicator\n\tstartPos  replication.Position\n\tstopPos   replication.Position\n\tsaveStop  bool\n\tcopyState map[string]*sqltypes.Result\n\n\treplicatorPlan *ReplicatorPlan\n\ttablePlans     map[string]*TablePlan\n","sourceCodeStart":35,"sourceCodeEnd":71,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vttablet/tabletmanager/vreplication/vplayer.go#L35-L71","documentation":"errVPlayerStalled is returned when the vplayer (binlog player) has made no replication progress for vplayerProgressDeadline (5 minutes). It signals that transactions from the source are being applied but take abnormally long, so vreplication is effectively stalled.","triggerScenarios":"Send/recordHeartbeat/TestRelayLogSendStallDeferredWhileThrottled observe no heartbeat/progress advance within 5 minutes while applying transactions on the target mysqld.","commonSituations":"Long-running or unindexed DMLs on the target (missing primary/secondary keys causing full table scans); target mysqld overloaded, locked, or disk-starved; very large single transactions from the source; throttling combined with slow apply.","solutions":["Run EXPLAIN on the queries being replicated (check vreplication logs for the stalled transaction) and add missing indexes/primary keys on the target tables","Check target mysqld health: slow query log, locks (SHOW PROCESSLIST / INNODB STATUS), disk I/O, and replication lag","Reduce transaction size on the source (break up huge batch DMLs) and verify throttler settings are not starving the vreplication stream"],"exampleFix":"// before (target table missing index, DML full-scans)\nCREATE TABLE t (id INT, name VARCHAR(64));\n// after\nCREATE TABLE t (id INT PRIMARY KEY, name VARCHAR(64), KEY idx_name (name));","handlingStrategy":"retry","validationCode":"// Before/while vreplication runs, check target health and slow queries:\n// SHOW PROCESSLIST — long-running queries on vreplication tables\n// SELECT * FROM information_schema.innodb_trx WHERE trx_started < NOW() - INTERVAL 5 MINUTE","typeGuard":null,"tryCatchPattern":"if err := stream.Send(ctx); err != nil {\n    if strings.Contains(err.Error(), \"progress stalled\") {\n        log.Warn(\"vreplication stalled; checking target health\", slog.Any(\"error\", err))\n        // alert + inspect EXPLAIN of the stalled txn, then let vreplication retry\n    }\n    return err\n}","preventionTips":["Ensure every replicated table has a primary key and appropriate indexes","Break up huge batch DMLs on the source","Monitor target mysqld for locks, load, and disk pressure","Keep throttler settings from starving vreplication"],"tags":["vreplication","vplayer","stall","performance"],"backgroundTag":"replication-stalled","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}