influxdata/influxdb · error · HeapDumpError

blocking task join

Error message

blocking task join: {0}

What it means

HeapDumpError::Join wraps a tokio::task::JoinError produced when the spawn_blocking task that performs the jemalloc dump is joined by dump_heap_profile and the task failed — either it panicked or the runtime is shutting down. The dump itself may never have completed; this variant reports the async task machinery failure, with the original JoinError available via #[from].

Solutions

1. Avoid issuing heap dumps during shutdown; guard the endpoint so it rejects once shutdown begins
2. Inspect the wrapped JoinError (is_panic/is_cancelled) to distinguish panic vs cancellation and fix the underlying panic if present
3. Ensure dump_heap_profile is always called within an active Tokio runtime

Example fix

// before
let stream = dump_heap_profile().await?;
// after
match dump_heap_profile().await {
    Err(HeapDumpError::Join(e)) if e.is_cancelled() => return Err(StatusCode::SERVICE_UNAVAILABLE.into()),
    other => other?,
}

match dump_heap_profile().await {
    Err(HeapDumpError::Join(e)) if e.is_cancelled() => /* runtime shutting down; skip */,
    Err(HeapDumpError::Join(e)) => tracing::error!(%e, "dump task panicked"),
    other => other?,
}

When it happens

Trigger: Calling dump_heap_profile() during Tokio runtime shutdown (task cancelled), or when the blocking task panics internally (e.g. jemalloc aborting).

Common situations: Graceful shutdown racing an in-flight pprof heap dump request; a panic inside the blocking dump path; running dump_heap_profile outside a Tokio runtime context.

Related errors

• Error initializing tokio runtime
• heap profiling is disabled; restart with
• jemalloc prof.dump failed
• join error
• snapshot manifest parse task panicked

AI-assisted analysis of influxdata/influxdb@06200ef96b (2026-09-19). Data as JSON: /api/errors/6e7d4bf78a20cc58. Report an issue: GitHub.

Appendix: source

Thrown at core/jemalloc_stats/src/lib.rs:100

    target_env = "musl",
)) {
    "_RJEM_MALLOC_CONF"
} else {
    "MALLOC_CONF"
};

/// Errors returned by [`dump_heap_profile`].
#[derive(Debug, thiserror::Error)]
pub enum HeapDumpError {
    #[error("tempfile path was not valid C string (non-utf8 or interior NUL)")]
    BadTempPath,
    #[error("heap profiling is disabled; restart with {MALLOC_CONF_ENV}=prof:true to enable")]
    ProfilingDisabled,
    #[error("jemalloc prof.dump failed: {0}")]
    Jemalloc(#[from] tikv_jemalloc_ctl::Error),
    #[error("tempfile i/o: {0}")]
    Io(#[from] std::io::Error),
    #[error("blocking task join: {0}")]
    Join(#[from] tokio::task::JoinError),
}

/// Trigger a jemalloc heap profile dump and return a streaming reader for
/// the resulting `heap_v2` bytes.
///
/// Requires jemalloc built with `--enable-prof` (i.e. `tikv-jemallocator`
/// `profiling` feature) AND the binary started with `prof:true` in the
/// platform-appropriate `MALLOC_CONF` env var (or via a baked-in
/// `malloc_conf` static — see [`DEFAULT_MALLOC_CONF`]). When profiling is
/// disabled at startup this returns [`HeapDumpError::ProfilingDisabled`].
///
/// The function handles all file machinery internally: it allocates a
/// tempfile, runs jemalloc's synchronous `prof.dump`, unlinks the path, and
/// hands back an open fd wrapped in a [`HeapProfileStream`]. The tempfile
/// is unlinked before this call returns — POSIX semantics keep the inode
/// alive via the open fd, so no profile bytes remain on disk after the
/// stream is dropped.

View on GitHub (pinned to 06200ef96b)