temporalio/temporal · error

transaction rollback failed: %w

Error message

transaction rollback failed: %w

What it means

In InsertIntoVisibility, a deferred rollback runs for every exit path; if tx.Rollback() fails with anything other than sql.ErrTxDone (which just means the tx was already committed/closed), the function's return error is overwritten with 'transaction rollback failed: %w'. Per the in-code comment this should never happen unless the DB connection was lost. Note the wrapped variable is retError, so the original failure (if any) is carried inside the wrap chain.

Source

Thrown at common/persistence/sql/sqlplugin/mysql/visibility.go:109

	finalRow := mdb.prepareRowForDB(row)
	defer func() {
		retError = mdb.handle.ConvertError(retError)
	}()
	db, err := mdb.handle.DB()
	if err != nil {
		return nil, err
	}

	tx, err := db.BeginTxx(ctx, nil)
	if err != nil {
		return nil, err
	}
	defer func() {
		err := tx.Rollback()
		// If the error is sql.ErrTxDone, it means the transaction already closed, so ignore error.
		if err != nil && !errors.Is(err, sql.ErrTxDone) {
			// Transaction rollback error should never happen, unless db connection was lost.
			retError = fmt.Errorf("transaction rollback failed: %w", retError)
		}
	}()
	result, err = tx.NamedExecContext(ctx, templateInsertWorkflowExecution, finalRow)
	if err != nil {
		return nil, fmt.Errorf("unable to insert workflow execution: %w", err)
	}
	_, err = tx.NamedExecContext(ctx, templateInsertCustomSearchAttributes, finalRow)
	if err != nil {
		return nil, fmt.Errorf("unable to insert custom search attributes: %w", err)
	}
	_, err = tx.NamedExecContext(ctx, templateInsertChasmSearchAttributes, finalRow)
	if err != nil {
		return nil, fmt.Errorf("unable to insert chasm search attributes: %w", err)
	}
	err = tx.Commit()
	if err != nil {
		return nil, err
	}

View on GitHub (pinned to bde624efd1)

Solutions

1. Check the wrapped cause (%w chain) with errors.Unwrap / %v logging to see 'bad connection' or 'driver: bad connection'.
2. Confirm MySQL connectivity and inspect server logs around the incident time for restarts or killed connections.
3. Tune MySQL wait_timeout / proxy idle timeouts to exceed worst-case transaction durations.
4. Retry the visibility insert — a fresh transaction on a healthy pooled connection should succeed.
5. If frequent, enable driver-side connection liveness checks (e.g. connMaxLifetime shorter than server timeout).

if err := db.InsertIntoVisibility(ctx, row); err != nil {
    if strings.Contains(err.Error(), "transaction rollback failed") {
        logger.Warn("connection lost during visibility insert tx; retrying", tag.Error(err))
        return retryOp(ctx) // bounded retry with backoff
    }
    return err
}

When it happens

Trigger: InsertIntoVisibility: deferred tx.Rollback() returns a non-ErrTxDone error — the MySQL connection backing the transaction dropped or was killed between BeginTxx and the deferred rollback.

Common situations: MySQL server restart or failover mid-insert; connection killed by wait_timeout or proxy idle timeout; network partition between history service and MySQL; DBA killing a long-running connection.

Related errors

• unable to delete custom search attributes: %w
• unable to delete workflow execution: %w
• unable to delete chasm search attributes: %w
• rowsAffected returned error when initializing DLQ metadata
• ErrQueueAlreadyExists

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/2420f46c8e8f5ad7. Report an issue: GitHub.