What causes "CancellableTask aborted due to shutdown timeout" in LemmyNet/lemmy?

Increase the shutdown timeout passed to CancellableTask so slow tasks have time to finish

CancellableTask aborted due to shutdown timeout

Error message

CancellableTask aborted due to shutdown timeout

What it means

CancellableTask wraps an async task with a cancellation future; this error is returned when the task did not finish after the shutdown/cancel signal within the given timeout, so the underlying tokio task was aborted. It signals a graceful-shutdown failure rather than a task-level business error.

Solutions

Increase the shutdown timeout passed to CancellableTask so slow tasks have time to finish
Make the wrapped task respond promptly to the abort/cancellation signal (avoid long non-cancellable awaits)
Check why the task hung — e.g. stuck DB connections or network calls without timeouts
Retry/inspect at restart; the task was force-aborted so any in-progress work was dropped

Example fix

// before
CancellableTask::new(task, Duration::from_secs(1))
// after
CancellableTask::new(task, Duration::from_secs(30)) // allow graceful drain

Defensive patterns

Strategy: retry

Validate before calling

// ensure the task is cancellation-aware before wrapping
// e.g. select! on a shutdown token inside the task loop

Try / catch

match cancellable_task.join().await {
  Err(e) if e.to_string().contains("aborted due to shutdown timeout") => {
    warn!("graceful shutdown timed out; task was force-aborted");
    // retry or continue shutdown
  }
  other => other?,
}

Prevention

Set a shutdown timeout larger than your task's worst-case step
Use cancellation tokens inside long loops and check them per iteration
Avoid unbounded blocking I/O inside cancellable tasks
Test shutdown under load before deploying

When it happens

Trigger: Shutting down the federation worker/send loop while the inner task is still running and does not react to cancellation (or is blocked on a slow I/O call) longer than the configured timeout duration.

Common situations: Instance restart/redeploy while federation tasks are mid-flight; long blocking DB or HTTP calls inside the wrapped task; a bug in the task that ignores the cancellation token; timeout configured too aggressively low.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

AI-assisted analysis of LemmyNet/lemmy@439734dd63 (2026-09-06). Data as JSON: /api/errors/240a222ad505e829. Report an issue: GitHub.

Appendix: source

Thrown at crates/apub/send/src/util.rs:120

        if stop2.is_cancelled() {
          return;
        } else {
          tracing::warn!("task exited, restarting: {res:?}");
        }
      }
    });
    let abort = task.abort_handle();
    CancellableTask {
      f: Box::pin(async move {
        stop.cancel();
        tokio::select! {
            r = task => {
              r.context("CancellableTask failed to cancel cleanly, returned error")?;
              Ok(())
            },
            _ = sleep(timeout) => {
                abort.abort();
                Err(anyhow!("CancellableTask aborted due to shutdown timeout"))
            }
        }
      }),
    }
  }

  /// cancel the cancel signal, wait for timeout for the task to stop gracefully, otherwise abort it
  pub async fn cancel(self) -> Result<(), anyhow::Error> {
    self.f.await
  }
}

/// assuming apub priv key and ids are immutable, then we don't need to have TTL
/// TODO: capacity should be configurable maybe based on memory use
pub(crate) async fn get_actor_cached(
  pool: &mut DbPool<'_>,
  actor_type: ActorType,
  actor_apub_id: &Url,

View on GitHub (pinned to 439734dd63)