timed out while transferring retained messages from '$MQTT.r

Error message

timed out while transferring retained messages from '$MQTT.rmsgs' after %v, %d processed, %d successfully transferred

What it means

Raised when migrating old retained messages from the legacy '$MQTT.rmsgs' stream exceeds the configured deadline. The server logs a Notice and aborts the transfer, reporting how many messages were processed and how many were successfully transferred so far.

Source

Thrown at server/mqtt.go:3321

			subject := mqttRetainedMsgsStreamSubject + rmsg.Subject
			if _, err = jsa.storeMsg(subject, 0, smsg.Data); err != nil {
				log.Errorf("    Unable to transfer the retained message with sequence %d: %v", smsg.Sequence, err)
			}
			transferred++
		} else {
			log.Warnf("    Unable to unmarshal retained message with sequence %d, skipping", smsg.Sequence)
		}

		// Delete the original message.
		if err := jsa.deleteMsg(mqttRetainedMsgsStreamName, smsg.Sequence, true); err != nil {
			log.Errorf("    Unable to clean up the retained message with sequence %d: %v", smsg.Sequence, err)
			return err
		}
		processed++

		now := time.Now()
		if now.After(deadline) {
			err := fmt.Errorf("timed out while transferring retained messages from '$MQTT.rmsgs' after %v, %d processed, %d successfully transferred", now.Sub(start), processed, transferred)
			log.Noticef(err.Error())
			return err
		}
	}
	if processed > 0 {
		log.Noticef("Processed %d messages from '$MQTT.rmsgs', successfully transferred %d in %v", processed, transferred, time.Since(start))
	} else {
		log.Debugf("No messages found to transfer from '$MQTT.rmsgs'")
	}
	return nil
}

func (as *mqttAccountSessionManager) getCachedRetainedMsg(subject string) *mqttRetainedMsg {
	v, ok := as.rmsCache.Load(subject)
	if !ok {
		return nil
	}
	rm := v.(*mqttRetainedMsg)

View on GitHub (pinned to 3a66a489d2)

Solutions

Increase the retained-message migration timeout/limit in server config
Run the migration when the broker is idle and storage is fast
Re-run/restart the server to resume/complete the remaining transfer
Trim or expire unnecessary retained messages before upgrading

Example fix

// before
mqtt { retained_msgs_migration_timeout: "5s" }
// after
mqtt { retained_msgs_migration_timeout: "5m" }

Defensive patterns

Strategy: fallback

Validate before calling

// Before upgrade, count legacy retained messages to size the timeout
const count = await estimateLegacyRetainedCount()
const timeout = Math.max(config.migrationTimeout, count * 50 /*ms*/ + '30s')

Try / catch

// Detect partial migration in logs and re-run server to finish
server.on('log', m => {
  if (m.includes("timed out while transferring retained messages")) {
    scheduleServerRestartToResumeMigration()
  }
})

Prevention

Estimate retained-message volume before upgrading and size the timeout accordingly
Run migrations during low-traffic windows
Purge stale retained messages before upgrade
Re-run the server after a timed-out migration to finish the transfer

When it happens

Trigger: Server upgrade path where retained messages are being copied from '$MQTT.rmsgs' to the new storage; the loop's per-iteration time check finds now.After(deadline) while many messages remain.

Common situations: Very large number of retained messages after an upgrade; slow JetStream during migration; server under heavy load or with slow disk while starting up with a migration deadline configured.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/66bc966eac4c958f. Report an issue: GitHub.