{"record":{"id":"ff224d249e1bc7cd","repo":"ory/hydra","slug":"failed-to-run-all-sql-migrations-direction-s-tim","errorCode":null,"errorMessage":"failed to run all SQL migrations: direction=%s timeout=%s","messagePattern":"failed to run all SQL migrations: direction=(.+?) timeout=(.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"oryx/popx/migrator.go","lineNumber":485,"sourceCode":"\t\t},\n\t}\n\n\tif err := mb.createMigrationStatusTableTransaction(ctx, workload...); err != nil {\n\t\treturn errors.WithStack(err)\n\t}\n\n\tl.WithField(\"migration_table\", mtn).Debug(\"Successfully migrated legacy schema_migration to new transactional schema_migration table.\")\n\n\treturn nil\n}\n\nfunc (mb *MigrationBox) isolatedTransaction(ctx context.Context, direction string, fn func(c *pop.Connection) error) (err error) {\n\tctx, span := startSpan(ctx, MigrationRunTransactionOpName, trace.WithAttributes(attribute.String(\"migration_direction\", direction)))\n\tdefer otelx.End(span, &err)\n\n\tif mb.perMigrationTimeout > 0 {\n\t\tvar cancel context.CancelFunc\n\t\tctx, cancel = context.WithTimeoutCause(ctx, mb.perMigrationTimeout, errors.Errorf(\"failed to run all SQL migrations: direction=%s timeout=%s\", direction, mb.perMigrationTimeout))\n\t\tdefer cancel()\n\t}\n\n\treturn Transaction(ctx, mb.c.WithContext(ctx), func(ctx context.Context, connection *pop.Connection) error {\n\t\treturn errors.WithStack(fn(connection))\n\t})\n}\n\nfunc (mb *MigrationBox) createMigrationStatusTableTransaction(ctx context.Context, transactions ...[]string) error {\n\tfor _, statements := range transactions {\n\t\t// CockroachDB does not support transactional schema changes, so we have to run\n\t\t// the statements outside of a transaction. The same applies to any dialect\n\t\t// that opts into autocommit DDL (see noTxDDL).\n\t\tif mb.noTxDDL() {\n\t\t\tfor _, statement := range statements {\n\t\t\t\tif err := mb.c.WithContext(ctx).RawQuery(statement).Exec(); err != nil {\n\t\t\t\t\treturn errors.Wrapf(err, \"unable to execute statement: %s\", statement)\n\t\t\t\t}","sourceCodeStart":467,"sourceCodeEnd":503,"githubUrl":"https://github.com/ory/hydra/blob/4174065ffb052799890f7480f5360a877a67ffc1/oryx/popx/migrator.go#L467-L503","documentation":"This error is the context deadline cause attached when an isolated per-migration transaction exceeds the configured perMigrationTimeout. isolatedTransaction wraps each migration (or migration-init) run in its own transaction; when mb.perMigrationTimeout > 0 it installs context.WithTimeoutCause so that if the transaction does not finish in time, the cancellation cause 'failed to run all SQL migrations: direction=%s timeout=%s' surfaces instead of a bare 'context deadline exceeded'. It is the library's way of telling you a single migration ran longer than the allowed per-migration budget.","triggerScenarios":"Calling any MigrationBox operation that runs migrations through isolatedTransaction (status/run/createTransactionalMigrationTable/migrateToTransactionalMigrationTable via createMigrationStatusTableTransaction with direction 'init') while mb.perMigrationTimeout > 0, and the transaction's statements take longer than that timeout. Typically a long-running DDL statement (large ALTER TABLE, index creation on a big table) or a blocked/stalled connection.","commonSituations":"Running an expensive data-backfill or index build on a large production table; a database lock held by another session blocking DDL; setting perMigrationTimeout too aggressively (e.g. seconds) for a migration that legitimately takes minutes; slow network or an overloaded database server during CI.","solutions":["Increase the per-migration timeout (mb.perMigrationTimeout, set via migration box options) to comfortably cover your slowest migration.","Inspect which migration in the given direction is slow and optimize it (avoid long table rewrites, add indexes outside the migration, batch data changes).","Check for blocking locks or long-running transactions on the database (pg_stat_activity / SHOW PROCESSLIST) and resolve contention.","Set perMigrationTimeout to 0 to disable the per-migration timeout if you manage timeouts at a different level."],"exampleFix":"// before\nmb, err := popx.NewMigrationBox(migrations, popx.WithPerMigrationTimeout(30*time.Second), c)\n// after\nmb, err := popx.NewMigrationBox(migrations, popx.WithPerMigrationTimeout(10*time.Minute), c)","handlingStrategy":"retry","validationCode":"// before running migrations, ensure the timeout covers the workload\nif mb.PerMigrationTimeout() > 0 && mb.PerMigrationTimeout() < 5*time.Minute {\n    log.Printf(\"per-migration timeout %s may be too low for large DDL\", mb.PerMigrationTimeout())\n}","typeGuard":null,"tryCatchPattern":"err := mb.MigrateUp(ctx)\nif err != nil && strings.Contains(err.Error(), \"failed to run all SQL migrations\") && strings.Contains(err.Error(), \"timeout=\") {\n    // per-migration timeout hit: retry with a larger timeout budget\n    return retryWithBiggerTimeout(err)\n}","preventionTips":["Set perMigrationTimeout generously (minutes, not seconds) for production databases.","Test large migrations against a production-sized dataset before deploying.","Monitor for blocking locks (pg_stat_activity, SHOW PROCESSLIST) during deploys.","Split long data migrations into smaller batched steps."],"tags":["database","migration","timeout","transaction"],"backgroundTag":"migration-timeout","analyzedSha":"4174065ffb052799890f7480f5360a877a67ffc1","analyzedAt":"2026-09-03T14:52:41.581Z","contentChangedAt":"2026-09-03T14:52:41.581Z","schemaVersion":2},"datasetVersion":"2026-09-10T17:17:09.494Z"}