{"record":{"id":"186c8692026e6db4","repo":"RocketChat/Rocket.Chat","slug":"not-migrating-control-is-locked-will-retry","errorCode":null,"errorMessage":"Not migrating, control is locked. Will retry.","messagePattern":"Not migrating, control is locked\\. Will retry\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"apps/meteor/server/lib/migrations.ts","lineNumber":190,"sourceCode":"\t}\n\n\t// version 0 means it is a fresh database, just set the control to latest known version and skip\n\tif (currentVersion === 0) {\n\t\tawait setControl({\n\t\t\tlocked: false,\n\t\t\tversion: orderedMigrations[orderedMigrations.length - 1].version,\n\t\t});\n\t\treturn true;\n\t}\n\n\tconst version = targetVersion === 'latest' ? orderedMigrations[orderedMigrations.length - 1].version : targetVersion;\n\n\t// get latest version\n\t// const { version } = orderedMigrations[orderedMigrations.length - 1];\n\n\tif (!(await lock())) {\n\t\tif (currentAttempt <= maxAttempts) {\n\t\t\tlog.warn({\n\t\t\t\tmsg: 'Not migrating, control is locked. Will retry.',\n\t\t\t\tretryIntervalSeconds: retryInterval,\n\t\t\t\tattempt: currentAttempt,\n\t\t\t\tmaxAttempts,\n\t\t\t});\n\n\t\t\tawait sleep(retryInterval * 1000);\n\n\t\t\tcurrentAttempt++;\n\t\t\treturn migrateDatabase(targetVersion, subcommands);\n\t\t}\n\t\tconst control = await getControl(); // Side effect: upserts control document.\n\t\tshowErrorBox(\n\t\t\t'ERROR! SERVER STOPPED',\n\t\t\t[\n\t\t\t\t'Your database migration control is locked.',\n\t\t\t\t'Please make sure you are running the latest version and try again.',\n\t\t\t\t'If the problem persists, please contact support.',","sourceCodeStart":172,"sourceCodeEnd":208,"githubUrl":"https://github.com/RocketChat/Rocket.Chat/blob/b2c16d5842cbe6b69b59bdf6fc5e5f1afcd1f0b0/apps/meteor/server/lib/migrations.ts#L172-L208","documentation":"Database migrations are guarded by a lock in the control document so only one instance migrates at a time. This warning means lock() failed — another instance holds the lock, or a crashed instance left control.locked=true. The server sleeps retryInterval (10s) and retries recursively, up to maxAttempts (30, ~5 minutes), after which it halts with 'ERROR! SERVER STOPPED — Database migration control is locked'.","triggerScenarios":"Booting multiple Rocket.Chat instances against the same database at a new version simultaneously (scale-up, k8s rollout, docker-compose scale); a previous migration that crashed mid-run leaving locked=true; an instance stuck mid-migration on a slow or unhealthy MongoDB.","commonSituations":"Rolling deployments where all replicas start at once; a server killed during migration (OOM, restart) so the lock was never released; dev and prod pointing at the same DB; replica set step-down during migration.","solutions":["Do nothing if another instance is legitimately migrating — this instance retries every 10s and proceeds once the lock releases","If no other instance is running, clear the stale lock: in MongoDB set the migrations control document's locked=false (back up the database first)","For rolling deployments, let the first instance finish booting before starting the rest","If the server stops after ~30 attempts, inspect what holds the lock (another live instance? Mongo health?) and restart after resolving"],"exampleFix":"// mongo shell, only when no instance is migrating\n// before\ndb.rocketchat_migrations.findOne() // { locked: true, version: 355, ... }\n\n// after\ndb.rocketchat_migrations.updateOne({}, { $set: { locked: false } })","handlingStrategy":"retry","validationCode":"// Pre-flight before starting an instance that will migrate\nconst control = db.collection('rocketchat_migrations').findOne();\nif (control?.locked) {\n  // find out who holds the lock; only clear it after confirming no instance is migrating\n  logger.warn(`Migration lock held (version ${control.version}); waiting instead of forcing`);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Deploy sequentially: let the first instance complete migrations before scaling up","Never kill a server mid-migration; if it dies, verify and clear control.locked manually with a DB backup","Keep MongoDB healthy (replica set, no step-downs) during migration windows","Expect up to 30 retries × 10s before the server halts — treat repeated warnings as an operator alert, not noise"],"tags":["migrations","database","startup","distributed-lock","mongodb"],"backgroundTag":"migration-lock-contention","analyzedSha":"b2c16d5842cbe6b69b59bdf6fc5e5f1afcd1f0b0","analyzedAt":"2026-08-18T15:26:39.429Z","contentChangedAt":"2026-08-18T15:26:39.429Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}