{"record":{"id":"66bc966eac4c958f","repo":"nats-io/nats-server","slug":"timed-out-while-transferring-retained-messages-fro","errorCode":null,"errorMessage":"timed out while transferring retained messages from '$MQTT.rmsgs' after %v, %d processed, %d successfully transferred","messagePattern":"timed out while transferring retained messages from '\\$MQTT\\.rmsgs' after (.+?), (.+?) processed, (.+?) successfully transferred","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"server/mqtt.go","lineNumber":3321,"sourceCode":"\t\t\tsubject := mqttRetainedMsgsStreamSubject + rmsg.Subject\n\t\t\tif _, err = jsa.storeMsg(subject, 0, smsg.Data); err != nil {\n\t\t\t\tlog.Errorf(\"    Unable to transfer the retained message with sequence %d: %v\", smsg.Sequence, err)\n\t\t\t}\n\t\t\ttransferred++\n\t\t} else {\n\t\t\tlog.Warnf(\"    Unable to unmarshal retained message with sequence %d, skipping\", smsg.Sequence)\n\t\t}\n\n\t\t// Delete the original message.\n\t\tif err := jsa.deleteMsg(mqttRetainedMsgsStreamName, smsg.Sequence, true); err != nil {\n\t\t\tlog.Errorf(\"    Unable to clean up the retained message with sequence %d: %v\", smsg.Sequence, err)\n\t\t\treturn err\n\t\t}\n\t\tprocessed++\n\n\t\tnow := time.Now()\n\t\tif now.After(deadline) {\n\t\t\terr := fmt.Errorf(\"timed out while transferring retained messages from '$MQTT.rmsgs' after %v, %d processed, %d successfully transferred\", now.Sub(start), processed, transferred)\n\t\t\tlog.Noticef(err.Error())\n\t\t\treturn err\n\t\t}\n\t}\n\tif processed > 0 {\n\t\tlog.Noticef(\"Processed %d messages from '$MQTT.rmsgs', successfully transferred %d in %v\", processed, transferred, time.Since(start))\n\t} else {\n\t\tlog.Debugf(\"No messages found to transfer from '$MQTT.rmsgs'\")\n\t}\n\treturn nil\n}\n\nfunc (as *mqttAccountSessionManager) getCachedRetainedMsg(subject string) *mqttRetainedMsg {\n\tv, ok := as.rmsCache.Load(subject)\n\tif !ok {\n\t\treturn nil\n\t}\n\trm := v.(*mqttRetainedMsg)","sourceCodeStart":3303,"sourceCodeEnd":3339,"githubUrl":"https://github.com/nats-io/nats-server/blob/3a66a489d262bf89b71a71c955c94920394532f3/server/mqtt.go#L3303-L3339","documentation":"Raised when migrating old retained messages from the legacy '$MQTT.rmsgs' stream exceeds the configured deadline. The server logs a Notice and aborts the transfer, reporting how many messages were processed and how many were successfully transferred so far.","triggerScenarios":"Server upgrade path where retained messages are being copied from '$MQTT.rmsgs' to the new storage; the loop's per-iteration time check finds now.After(deadline) while many messages remain.","commonSituations":"Very large number of retained messages after an upgrade; slow JetStream during migration; server under heavy load or with slow disk while starting up with a migration deadline configured.","solutions":["Increase the retained-message migration timeout/limit in server config","Run the migration when the broker is idle and storage is fast","Re-run/restart the server to resume/complete the remaining transfer","Trim or expire unnecessary retained messages before upgrading"],"exampleFix":"// before\nmqtt { retained_msgs_migration_timeout: \"5s\" }\n// after\nmqtt { retained_msgs_migration_timeout: \"5m\" }\n","handlingStrategy":"fallback","validationCode":"// Before upgrade, count legacy retained messages to size the timeout\nconst count = await estimateLegacyRetainedCount()\nconst timeout = Math.max(config.migrationTimeout, count * 50 /*ms*/ + '30s')","typeGuard":null,"tryCatchPattern":"// Detect partial migration in logs and re-run server to finish\nserver.on('log', m => {\n  if (m.includes(\"timed out while transferring retained messages\")) {\n    scheduleServerRestartToResumeMigration()\n  }\n})","preventionTips":["Estimate retained-message volume before upgrading and size the timeout accordingly","Run migrations during low-traffic windows","Purge stale retained messages before upgrade","Re-run the server after a timed-out migration to finish the transfer"],"tags":["mqtt","retained-messages","timeout","migration"],"backgroundTag":"migration-timeout","analyzedSha":"3a66a489d262bf89b71a71c955c94920394532f3","analyzedAt":"2026-09-02T04:41:54.247Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}