failed to run all SQL migrations: direction=%s timeout=%s

Error message

failed to run all SQL migrations: direction=%s timeout=%s

What it means

This error is the context deadline cause attached when an isolated per-migration transaction exceeds the configured perMigrationTimeout. isolatedTransaction wraps each migration (or migration-init) run in its own transaction; when mb.perMigrationTimeout > 0 it installs context.WithTimeoutCause so that if the transaction does not finish in time, the cancellation cause 'failed to run all SQL migrations: direction=%s timeout=%s' surfaces instead of a bare 'context deadline exceeded'. It is the library's way of telling you a single migration ran longer than the allowed per-migration budget.

Source

Thrown at oryx/popx/migrator.go:485

		},
	}

	if err := mb.createMigrationStatusTableTransaction(ctx, workload...); err != nil {
		return errors.WithStack(err)
	}

	l.WithField("migration_table", mtn).Debug("Successfully migrated legacy schema_migration to new transactional schema_migration table.")

	return nil
}

func (mb *MigrationBox) isolatedTransaction(ctx context.Context, direction string, fn func(c *pop.Connection) error) (err error) {
	ctx, span := startSpan(ctx, MigrationRunTransactionOpName, trace.WithAttributes(attribute.String("migration_direction", direction)))
	defer otelx.End(span, &err)

	if mb.perMigrationTimeout > 0 {
		var cancel context.CancelFunc
		ctx, cancel = context.WithTimeoutCause(ctx, mb.perMigrationTimeout, errors.Errorf("failed to run all SQL migrations: direction=%s timeout=%s", direction, mb.perMigrationTimeout))
		defer cancel()
	}

	return Transaction(ctx, mb.c.WithContext(ctx), func(ctx context.Context, connection *pop.Connection) error {
		return errors.WithStack(fn(connection))
	})
}

func (mb *MigrationBox) createMigrationStatusTableTransaction(ctx context.Context, transactions ...[]string) error {
	for _, statements := range transactions {
		// CockroachDB does not support transactional schema changes, so we have to run
		// the statements outside of a transaction. The same applies to any dialect
		// that opts into autocommit DDL (see noTxDDL).
		if mb.noTxDDL() {
			for _, statement := range statements {
				if err := mb.c.WithContext(ctx).RawQuery(statement).Exec(); err != nil {
					return errors.Wrapf(err, "unable to execute statement: %s", statement)
				}

View on GitHub (pinned to 4174065ffb)

Solutions

Increase the per-migration timeout (mb.perMigrationTimeout, set via migration box options) to comfortably cover your slowest migration.
Inspect which migration in the given direction is slow and optimize it (avoid long table rewrites, add indexes outside the migration, batch data changes).
Check for blocking locks or long-running transactions on the database (pg_stat_activity / SHOW PROCESSLIST) and resolve contention.
Set perMigrationTimeout to 0 to disable the per-migration timeout if you manage timeouts at a different level.

Example fix

// before
mb, err := popx.NewMigrationBox(migrations, popx.WithPerMigrationTimeout(30*time.Second), c)
// after
mb, err := popx.NewMigrationBox(migrations, popx.WithPerMigrationTimeout(10*time.Minute), c)

Defensive patterns

Strategy: retry

Validate before calling

// before running migrations, ensure the timeout covers the workload
if mb.PerMigrationTimeout() > 0 && mb.PerMigrationTimeout() < 5*time.Minute {
    log.Printf("per-migration timeout %s may be too low for large DDL", mb.PerMigrationTimeout())
}

Try / catch

err := mb.MigrateUp(ctx)
if err != nil && strings.Contains(err.Error(), "failed to run all SQL migrations") && strings.Contains(err.Error(), "timeout=") {
    // per-migration timeout hit: retry with a larger timeout budget
    return retryWithBiggerTimeout(err)
}

Prevention

Set perMigrationTimeout generously (minutes, not seconds) for production databases.
Test large migrations against a production-sized dataset before deploying.
Monitor for blocking locks (pg_stat_activity, SHOW PROCESSLIST) during deploys.
Split long data migrations into smaller batched steps.

When it happens

Trigger: Calling any MigrationBox operation that runs migrations through isolatedTransaction (status/run/createTransactionalMigrationTable/migrateToTransactionalMigrationTable via createMigrationStatusTableTransaction with direction 'init') while mb.perMigrationTimeout > 0, and the transaction's statements take longer than that timeout. Typically a long-running DDL statement (large ALTER TABLE, index creation on a big table) or a blocked/stalled connection.

Common situations: Running an expensive data-backfill or index build on a large production table; a database lock held by another session blocking DDL; setting perMigrationTimeout too aggressively (e.g. seconds) for a migration that legitimately takes minutes; slow network or an overloaded database server during CI.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of ory/hydra@4174065ffb (2026-09-03). Data as JSON: /api/errors/ff224d249e1bc7cd. Report an issue: GitHub.