vitessio/vitess · warning
error %v updating time
Error message
error %v updating time
What it means
updateHeartbeatTime writes the current timestamp into _vt.vreplication.time_updated (GenerateUpdateHeartbeat). Failure is wrapped in this error. The heartbeat tells operators and auto-tuning logic that the stream is alive; failure to write it usually means the tablet cannot talk to MySQL or the row vanished.
Source
Thrown at go/vt/vttablet/tabletmanager/vreplication/vreplicator.go:746
update, err := binlogplayer.GenerateUpdateTimeThrottled(vr.id, tm, appName, reasonThrottled)
if err != nil {
return err
}
if _, err := vr.dbClient.ExecuteFetch(update, maxRows); err != nil {
return fmt.Errorf("error %v updating time throttled", err)
}
return nil
})
return err
}
func (vr *vreplicator) updateHeartbeatTime(tm int64) error {
update, err := binlogplayer.GenerateUpdateHeartbeat(vr.id, tm)
if err != nil {
return err
}
if _, err := vr.dbClient.ExecuteFetch(update, maxRows); err != nil {
return fmt.Errorf("error %v updating time", err)
}
return nil
}
func (vr *vreplicator) clearFKCheck(dbClient *vdbClient) error {
_, err := dbClient.Execute("set @@session.foreign_key_checks=0")
return err
}
func (vr *vreplicator) clearFKRestrict(dbClient *vdbClient) error {
if !vr.needFKRestrict() {
return nil
}
_, err := dbClient.Execute("set @@session.restrict_fk_on_non_standard_key=0")
return err
}
func recalculatePKColsInfoByColumnNames(uniqueKeyColumnNames []string, colInfos []*ColumnInfo) (pkColInfos []*ColumnInfo) {View on GitHub (pinned to 01a25a7d17)
Solutions
- Inspect the wrapped MySQL error (connection lost vs. row missing vs. read-only).
- Check tablet-to-mysqld connectivity and recent failover/restart events.
- If the row is gone, the workflow was cancelled — re-create/restart the workflow rather than forcing writes.
- Monitor vitess_operation_errors / tablet logs for repeated heartbeat failures indicating a persistently unhealthy backend.
Defensive patterns
Strategy: retry
Validate before calling
// Health check before assuming stream liveness: // SELECT id, time_updated FROM _vt.vreplication WHERE id = <id>; // Verify backend reachability from the tablet (mysql ping).
Try / catch
if _, err := vr.dbClient.ExecuteFetch(update, maxRows); err != nil {
log.Warn("heartbeat update failed", slog.Any("error", err))
// retry with backoff; if row missing, the workflow was cancelled — stop
} Prevention
- Monitor time_updated staleness as a workflow-health metric.
- Alert on MySQL failovers and verify vreplication streams afterwards.
- Avoid concurrent workflow deletion; use vtctldclient ordering.
- Keep tablet-mysqld network paths redundant.
When it happens
Trigger: GenerateUpdateHeartbeat builds the UPDATE and ExecuteFetch fails: dead DB connection, mysqld down/read-only, or the _vt.vreplication row was deleted concurrently. Called from the heartbeat/recordHeartbeat path.
Common situations: MySQL failover or restart mid-stream; concurrent workflow cancellation; network partition between tablet and mysqld.
Related errors
- partial row image encountered: ensure binlog_row_image is se
- can't get charset to request binlog stream: %v
- error in processing binlog event %v
- error %v in writing recovery info %v
- could not set state: %v: %v
AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01).
Data as JSON: /api/errors/826e42253149fa5a.
Report an issue: GitHub.