{"record":{"id":"a3deeed97af74cea","repo":"LemmyNet/lemmy","slug":"next-id-to-send-is-not-as-expected","errorCode":null,"errorMessage":"{}: next id to send is not as expected: {:?} != {:?}","messagePattern":"(.+?): next id to send is not as expected: (.+?) != (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/apub/send/src/worker.rs","lineNumber":155,"sourceCode":"        self.handle_send_results().await?;\n        // handle_send_results does not guarantee that we are now in a condition where we want to\n        // send a new one, so repeat this check until the if no longer applies\n        continue;\n      }\n\n      // send a new activity if there is one\n      self.inbox_collector.update_communities().await?;\n      let next_id_to_send = ActivityId(last_sent_id.0 + 1);\n      let successfuls_len: i64 = self.successfuls.len().try_into()?;\n      {\n        // sanity check: calculate next id to send based on the last id and the in flight requests\n        let expected_next_id = self.state.last_successful_id.map(|last_successful_id| {\n          last_successful_id.0 + successfuls_len + i64::from(self.in_flight) + 1\n        });\n        // compare to next id based on incrementing\n        if expected_next_id != Some(next_id_to_send.0) {\n          return Err(\n            anyhow::anyhow!(\n              \"{}: next id to send is not as expected: {:?} != {:?}\",\n              self.instance.domain,\n              expected_next_id,\n              next_id_to_send\n            )\n            .into(),\n          );\n        }\n      }\n\n      let newest_id_opt = get_latest_activity_id(&mut self.pool()).await?;\n      let newest_id = newest_id_opt.unwrap_or(ActivityId(0));\n      if next_id_to_send > newest_id {\n        // If next id to send for this instance is higher than the highest sent_activity table id\n        // there may be a problem and activities wont send.\n        // However this can occur normally if there was no outgoing activity for a week\n        // and sent_activity was completely emptied by scheduled task.\n        if newest_id_opt.is_some() && next_id_to_send > ActivityId(newest_id.0 + 1) {","sourceCodeStart":137,"sourceCodeEnd":173,"githubUrl":"https://github.com/LemmyNet/lemmy/blob/439734dd638a2c06a2f907beab7dcf4646e88f86/crates/apub/send/src/worker.rs#L137-L173","documentation":"loop_until_stopped validates an internal invariant: the next activity id to send must equal last_successful_id + number of successful sends + in-flight count + 1. When the observed next id diverges from this computation, it means the local accounting of the send queue is out of sync with the database (e.g. rows deleted or ids skipped unexpectedly).","triggerScenarios":"Activities inserted/removed from sent_activities outside the worker's accounting (manual DB edits, deleted rows, unexpected serial gaps), or a bug in in-flight/last_successful_id bookkeeping while draining the queue.","commonSituations":"Operators manually delete rows from sent_activities; database restores/snapshots out of sync; crashes between insert and send creating gaps the invariant does not tolerate; long downtime with schema changes.","solutions":["Compare expected vs actual ids in the log to identify what changed in sent_activities","Check for manual deletions or external modifications of the sent_activities table","Restart the federation worker — state is reloaded from DB and the invariant recomputed","Verify no concurrent senders are running against the same database"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if let Err(e) = worker.init_and_loop().await {\n  error!(\"federation worker invariant violated, restarting worker: {e}\");\n  restart_worker(); // re-initializes state from DB\n}","preventionTips":["Never manually delete/modify sent_activities rows while workers run","Run only one federation sender per database","Investigate gaps between expected and actual next ids after crashes","Restore databases with workers stopped"],"tags":["federation","invariant","state","lemmy"],"backgroundTag":"internal-invariant-violation","analyzedSha":"439734dd638a2c06a2f907beab7dcf4646e88f86","analyzedAt":"2026-09-06T11:28:35.035Z","contentChangedAt":"2026-09-06T11:28:35.035Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}