LemmyNet/lemmy · error

{}: next id to send is not as expected: {:?} != {:?}

Error message

{}: next id to send is not as expected: {:?} != {:?}

What it means

loop_until_stopped validates an internal invariant: the next activity id to send must equal last_successful_id + number of successful sends + in-flight count + 1. When the observed next id diverges from this computation, it means the local accounting of the send queue is out of sync with the database (e.g. rows deleted or ids skipped unexpectedly).

Source

Thrown at crates/apub/send/src/worker.rs:155

        self.handle_send_results().await?;
        // handle_send_results does not guarantee that we are now in a condition where we want to
        // send a new one, so repeat this check until the if no longer applies
        continue;
      }

      // send a new activity if there is one
      self.inbox_collector.update_communities().await?;
      let next_id_to_send = ActivityId(last_sent_id.0 + 1);
      let successfuls_len: i64 = self.successfuls.len().try_into()?;
      {
        // sanity check: calculate next id to send based on the last id and the in flight requests
        let expected_next_id = self.state.last_successful_id.map(|last_successful_id| {
          last_successful_id.0 + successfuls_len + i64::from(self.in_flight) + 1
        });
        // compare to next id based on incrementing
        if expected_next_id != Some(next_id_to_send.0) {
          return Err(
            anyhow::anyhow!(
              "{}: next id to send is not as expected: {:?} != {:?}",
              self.instance.domain,
              expected_next_id,
              next_id_to_send
            )
            .into(),
          );
        }
      }

      let newest_id_opt = get_latest_activity_id(&mut self.pool()).await?;
      let newest_id = newest_id_opt.unwrap_or(ActivityId(0));
      if next_id_to_send > newest_id {
        // If next id to send for this instance is higher than the highest sent_activity table id
        // there may be a problem and activities wont send.
        // However this can occur normally if there was no outgoing activity for a week
        // and sent_activity was completely emptied by scheduled task.
        if newest_id_opt.is_some() && next_id_to_send > ActivityId(newest_id.0 + 1) {

View on GitHub (pinned to 439734dd63)

Solutions

  1. Compare expected vs actual ids in the log to identify what changed in sent_activities
  2. Check for manual deletions or external modifications of the sent_activities table
  3. Restart the federation worker — state is reloaded from DB and the invariant recomputed
  4. Verify no concurrent senders are running against the same database
Defensive patterns

Strategy: retry

Try / catch

if let Err(e) = worker.init_and_loop().await {
  error!("federation worker invariant violated, restarting worker: {e}");
  restart_worker(); // re-initializes state from DB
}

Prevention

When it happens

Trigger: Activities inserted/removed from sent_activities outside the worker's accounting (manual DB edits, deleted rows, unexpected serial gaps), or a bug in in-flight/last_successful_id bookkeeping while draining the queue.

Common situations: Operators manually delete rows from sent_activities; database restores/snapshots out of sync; crashes between insert and send creating gaps the invariant does not tolerate; long downtime with schema changes.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of LemmyNet/lemmy@439734dd63 (2026-09-06). Data as JSON: /api/errors/a3deeed97af74cea. Report an issue: GitHub.