mlflow/mlflow · critical · MlflowException

Database migration in unexpected state. Run manual upgrade.

Error message

Database migration in unexpected state. Run manual upgrade.

What it means

During store initialization the registry verifies that all expected tables (including SqlWebhookEvent) exist in the database. If any are missing, the schema is behind the code's expectations (migrations were not applied fully), and MLflow raises with instruction to run a manual upgrade instead of silently auto-migrating.

Source

Thrown at mlflow/store/model_registry/sqlalchemy_store.py:278

    def _dispose_engine(self):
        self.engine.dispose()

    @staticmethod
    def _verify_registry_tables_exist(engine):
        # Verify that all tables have been created.
        inspected_tables = set(sqlalchemy.inspect(engine).get_table_names())
        expected_tables = [
            SqlRegisteredModel.__tablename__,
            SqlModelVersion.__tablename__,
            SqlWebhook.__tablename__,
            SqlWebhookEvent.__tablename__,
        ]
        if any(table not in inspected_tables for table in expected_tables):
            # TODO: Replace the MlflowException with the following line once it's possible to run
            # the registry against a different DB than the tracking server:
            # mlflow.store.db.utils._initialize_tables(self.engine)
            raise MlflowException("Database migration in unexpected state. Run manual upgrade.")

    @staticmethod
    def _get_eager_registered_model_query_options():
        """
        A list of SQLAlchemy query options that can be used to eagerly
        load the following registered model attributes
        when fetching a registered model: ``registered_model_tags`` and
        ``registered_model_aliases``.
        """
        # Use a subquery load rather than a joined load in order to minimize the memory overhead
        # of the eager loading procedure. For more information about relationship loading
        # techniques, see https://docs.sqlalchemy.org/en/13/orm/
        # loading_relationships.html#relationship-loading-techniques
        return [
            sqlalchemy.orm.subqueryload(SqlRegisteredModel.registered_model_tags),
            sqlalchemy.orm.subqueryload(SqlRegisteredModel.registered_model_aliases),
        ]

View on GitHub (pinned to 6a27f2decc)

Solutions

Run `mlflow db upgrade <database-uri>` to apply pending alembic migrations
Verify the schema with `mlflow db verify <database-uri>`
Back up the database, then re-run migrations from a clean state
Pin the MLflow version to match the existing DB schema until you can migrate

Example fix

// before
mlflow server --backend-store-uri postgresql://...
// after
mlflow db upgrade postgresql://... && mlflow server --backend-store-uri postgresql://...

Defensive patterns

Strategy: validation

Validate before calling

import mlflow, subprocess
# As part of deployment:
subprocess.run(["mlflow", "db", "upgrade", db_uri], check=True)
subprocess.run(["mlflow", "db", "verify", db_uri], check=True)

Try / catch

from mlflow.exceptions import MlflowException
try:
    store = MlflowRegistryStore(db_uri)
except MlflowException as e:
    if 'Database migration in unexpected state' in str(e):
        run_db_upgrade(db_uri)  # mlflow db upgrade
        store = MlflowRegistryStore(db_uri)
    else:
        raise

Prevention

Always run `mlflow db upgrade <uri>` as a deployment step when bumping MLflow versions
Run `mlflow db verify` in CI against a copy of prod DB
Avoid restoring partial database dumps; restore complete backups
Pin MLflow versions consistently across server replicas

When it happens

Trigger: Pointing a newer MLflow server at an old database; a partially failed alembic migration; manually dropping tables; running the registry against a DB initialized by a different component.

Common situations: Version upgrades without running `mlflow db upgrade`; restoring half of a database dump; Docker images updated while a persisted volume holds an old schema; mixed-version server replicas.

Related errors

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/e1154b2af796958f. Report an issue: GitHub.