What causes "error %v updating time throttled" in vitessio/vitess?

Check the wrapped MySQL error for cause (dead connection vs. missing row vs. read-only).

error %v updating time throttled

Error message

error %v updating time throttled

What it means

When a stream is throttled, the code records the throttle event by updating the time_throttled column in _vt.vreplication (GenerateUpdateTimeThrottled builds the UPDATE). If that ExecuteFetch fails, this error wraps the MySQL error. It only concerns the throttling bookkeeping — the underlying data copy state is unaffected.

Source

Thrown at go/vt/vttablet/tabletmanager/vreplication/vreplicator.go:733

// with a rate limit so that it's only saved in the database at most once per
// throttleUpdatesRateLimiter.tickerTime.
// It also increments the throttled count in the stats to keep track of how many
// times a VReplication workflow, and the specific sub-component, is throttled by the
// tablet throttler over time. It also increments the global throttled count to keep
// track of how many times in total vreplication has been throttled across all workflows
// (both ones that currently exist and ones that no longer do).
func (vr *vreplicator) updateTimeThrottled(appThrottled throttlerapp.Name, reasonThrottled string) error {
	appName := appThrottled.String()
	vr.stats.ThrottledCounts.Add([]string{"tablet", appName}, 1)
	globalStats.ThrottledCount.Add(1)
	err := vr.throttleUpdatesRateLimiter.Do(func() error {
		tm := time.Now().Unix()
		update, err := binlogplayer.GenerateUpdateTimeThrottled(vr.id, tm, appName, reasonThrottled)
		if err != nil {
			return err
		}
		if _, err := vr.dbClient.ExecuteFetch(update, maxRows); err != nil {
			return fmt.Errorf("error %v updating time throttled", err)
		}
		return nil
	})
	return err
}

func (vr *vreplicator) updateHeartbeatTime(tm int64) error {
	update, err := binlogplayer.GenerateUpdateHeartbeat(vr.id, tm)
	if err != nil {
		return err
	}
	if _, err := vr.dbClient.ExecuteFetch(update, maxRows); err != nil {
		return fmt.Errorf("error %v updating time", err)
	}
	return nil
}

func (vr *vreplicator) clearFKCheck(dbClient *vdbClient) error {

View on GitHub (pinned to 01a25a7d17)

Solutions

Check the wrapped MySQL error for cause (dead connection vs. missing row vs. read-only).
Verify the _vt.vreplication row still exists for this id; if the workflow was cancelled, let the stream die instead of forcing updates.
Confirm target mysqld is writable and reachable; retry by restarting the workflow.
Check throttler health — if throttling is spurious, review throttler thresholds so this path stops firing.

Defensive patterns

Strategy: try-catch

Validate before calling

// Ensure throttling bookkeeping can succeed:
// SELECT id FROM _vt.vreplication WHERE id = <id>;
// SHOW GLOBAL VARIABLES LIKE 'read_only';

Try / catch

if _, err := vr.dbClient.ExecuteFetch(update, maxRows); err != nil {
	log.Warn("failed to record throttle time; data copy unaffected", slog.Any("error", err))
	// non-fatal: continue; investigate row/read-only only if persistent
}

Prevention

Treat throttle-bookkeeping errors as secondary health signals.
Keep target writable outside maintenance windows.
Check throttler configuration if throttle events fire unexpectedly often.
Don't delete workflow rows while streams are active.

When it happens

Trigger: The UPDATE of time_throttled fails at the point the throttler marks the stream as throttled: connection error, row deleted, read-only mysqld. Raised inside a throttler-client callback.

Common situations: Throttler fires during a MySQL restart/failover; concurrent workflow cancellation removed the row; read-only target during maintenance.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/f1274fc84703c1e5. Report an issue: GitHub.