What causes "database migration failed" in Netflix/chaosmonkey?

Read the wrapped cause (%+v) for the exact failing migration and SQL error

database migration failed

Error message

database migration failed

What it means

Migrate applies the embedded SQL migrations to the MySQL database using migrate.Exec. If any migration fails to execute, the error is wrapped as 'database migration failed'. The database is therefore left at whatever schema version it had, and the monkey will not start correctly.

Source

Thrown at mysql/mysql.go:452

	_, err = tx.Exec("INSERT INTO terminations (app, account, stack, cluster, region, asg, instance_id, killed_at, leashed) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
		i.AppName(), i.AccountName(), i.StackName(), i.ClusterName(), i.RegionName(), i.ASGName(), i.ID(), term.Time.In(time.UTC), term.Leashed)

	return err
}

var migrationSource = &migrate.AssetMigrationSource{
	Asset:    migration.Asset,
	AssetDir: migration.AssetDir,
	Dir:      "migration/mysql",
}

var databaseDialect = "mysql"

// Migrate upgrades a database to the latest database schema version.
func Migrate(mysqlDb MySQL) error {
	migrationCount, err := migrate.Exec(mysqlDb.db, databaseDialect, migrationSource, migrate.Up)
	if err != nil {
		return errors.Wrap(err, "database migration failed")
	}
	log.Println("Successfully applied database migrations. Number of migrations applied: ", migrationCount)

	return nil
}

View on GitHub (pinned to eaa28fb761)

Solutions

Read the wrapped cause (%+v) for the exact failing migration and SQL error
Verify the DB user has CREATE/ALTER/INSERT privileges on the target schema
Check the schema_migrations table for a partially applied migration and repair/re-run manually
Confirm the configured host/database is the intended one and reachable
Test migrations against a staging copy of the database first

Example fix

// before
GRANT SELECT, INSERT, UPDATE, DELETE ON chaosmonkey.* TO 'monkey'@'%';
// after
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, DROP ON chaosmonkey.* TO 'monkey'@'%';

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-check privileges and connectivity before Migrate
if err := mysqlDb.Ping(); err != nil {
    return fmt.Errorf("db unreachable before migration: %w", err)
}
// verify the user can DDL
txn, _ := mysqlDb.Begin()
if _, err := txn.Exec("CREATE TABLE IF NOT EXISTS _mig_probe (id int)"); err != nil {
    txn.Rollback()
    return fmt.Errorf("db user lacks DDL privileges: %w", err)
}
txn.Rollback()

Try / catch

if err := mysql.Migrate(store); err != nil {
    log.Printf("migration failed: %+v", err) // %+v reveals the failing migration/SQL
    return fmt.Errorf("startup aborted: %w", err)
}

Prevention

Run migrations in CI against a staging database before production deploys
Grant the monkey DB user full DDL rights on its schema
Inspect schema_migrations after any interrupted deployment
Back up the database before major version upgrades
Pin MySQL server versions known to work with the migration SQL

When it happens

Trigger: migrate.Exec returns an error: SQL syntax/schema incompatibility, the DB user lacks privileges to create/alter tables, connectivity drops mid-migration, or a prior failed migration left the schema version tracking inconsistent.

Common situations: First run against a fresh database with a restricted DB user; MySQL version incompatible with migration SQL; partially applied migration from an interrupted earlier run; wrong database pointed at by config.

Related errors

AI-assisted analysis of Netflix/chaosmonkey@eaa28fb761 (2026-09-03). Data as JSON: /api/errors/38ba7fedda4de551. Report an issue: GitHub.