vitessio/vitess · error

could not set state: %v: %v

Error message

could not set state: %v: %v

What it means

Thrown by setVReplicationState when the UPDATE that records the player's new state/message into _vt.vreplication fails to execute. The state transition itself is remembered in memory/blplStats, but persistence failed, so external observers may see a stale state. ApplyBinlogEvents calls this to mark the stream Stopped/Error.

Source

Thrown at go/vt/binlog/binlogplayer/binlog_player.go:553

	blp.position = position
	blp.blplStats.SetLastPosition(blp.position)
	if tx.EventToken.Timestamp != 0 {
		blp.blplStats.ReplicationLagSeconds.Store(now - tx.EventToken.Timestamp)
	}
	return nil
}

func (blp *BinlogPlayer) setVReplicationState(state binlogdatapb.VReplicationWorkflowState, message string) error {
	if message != "" {
		blp.blplStats.History.Add(&StatsHistoryRecord{
			Time:    time.Now(),
			Message: message,
		})
	}
	blp.blplStats.State.Store(state.String())
	query := fmt.Sprintf("update _vt.vreplication set state=%v, message=%v where id=%v", encodeString(state.String()), encodeString(MessageTruncate(message)), blp.uid)
	if _, err := blp.dbClient.ExecuteFetch(query, 1); err != nil {
		return fmt.Errorf("could not set state: %v: %v", query, err)
	}
	return nil
}

// VRSettings contains the settings of a vreplication table.
type VRSettings struct {
	StartPos           replication.Position
	StopPos            replication.Position
	MaxTPS             int64
	MaxReplicationLag  int64
	State              binlogdatapb.VReplicationWorkflowState
	WorkflowType       binlogdatapb.VReplicationWorkflowType
	WorkflowSubType    binlogdatapb.VReplicationWorkflowSubType
	WorkflowName       string
	DeferSecondaryKeys bool
	WorkflowOptions    *vtctldata.WorkflowOptions
}

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Fix the primary failure first (connection/row missing) — this error is usually a symptom.
2. Verify the row exists and grants allow UPDATE on _vt.vreplication.
3. Manually set the state if needed: `update _vt.vreplication set state='Error', message='...' where id=<uid>` via vtctldclient VReplicationExec.
4. Restart the workflow to re-establish a healthy connection and re-record state.

Example fix

// before: dead connection also blocks state write
//   (stream errors, setVReplicationState fails on same conn)
// after: restart stream to get fresh conn, state persists
//   vtctldclient Workflow Reshard <workflow> restart

// ensure the state row exists before transitions
var n int
targetDB.QueryRow("SELECT COUNT(*) FROM _vt.vreplication WHERE id = ?", uid).Scan(&n)
if n != 1 {
    return errors.New("cannot track stream state: _vt.vreplication row missing")
}

err := binlogplayer.ApplyBinlogEvents(ctx, blp)
if err != nil && strings.Contains(err.Error(), "could not set state") {
    // often a secondary failure: the primary error killed the connection
    log.Warn("state write failed; repairing row/state manually via vtctldclient")
    // vtctldclient VReplicationExec <keyspace> "update _vt.vreplication set state='Error' where id=<uid>"
}

When it happens

Trigger: blp.dbClient.ExecuteFetch(update query, 1) errors: connection closed (commonly because the stream is being torn down after the same failure), _vt.vreplication row deleted, insufficient grants, or target read-only — often a secondary failure layered on an earlier error.

Common situations: The stream is failing because the dbConn already died, so the state-write also fails; the vreplication row was removed concurrently; maintenance read_only window on the target.

Related errors

• could not set message: %v: %v
• could not set state: %v: %v
• partial row image encountered: ensure binlog_row_image is se
• can't get charset to request binlog stream: %v
• error in processing binlog event %v

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/cd670c9facab8957. Report an issue: GitHub.