influxdata/influxdb · critical

start watchdog thread

Error message

start watchdog thread

What it means

WatchdogConfig::install() spawns a dedicated OS thread ('tokio watchdog <name>') that periodically pings the tokio runtime to detect hangs. The .expect('start watchdog thread') panics when std::thread::Builder::spawn returns Err, i.e. the OS refused to create the thread. This is an environment/resource failure, not a logic bug: the thread limit (RLIMIT_NPROC / cgroup pids.max), memory exhaustion, or a forked child context typically prevents the spawn.

Source

Thrown at core/tokio_watchdog/src/lib.rs:181

                            let Some(d) = rx_response.blocking_recv() else {
                                return;
                            };
                            debug!(
                                runtime = runtime_name,
                                hang_secs = d.as_secs_f64(),
                                "tokio stops hanging",
                            );
                            d
                        }
                        Err(TryRecvError::Disconnected) => {
                            return;
                        }
                    };

                    metric_latency.record(d);
                }
            })
            .expect("start watchdog thread");
    }
}

#[cfg(test)]
mod tests {
    use std::sync::Arc;

    use test_helpers::tracing::TracingCapture;

    use super::*;

    #[tokio::test]
    #[should_panic(expected = "sum of tick and warn duration must be non-zero")]
    async fn test_panic_zero_duration() {
        let registry = Registry::default();
        WatchdogConfig::new(&Handle::current(), &registry)
            .with_tick_duration(Duration::ZERO)
            .with_warn_duration(Duration::ZERO)

View on GitHub (pinned to d28e26e048)

Solutions

1. Raise the process/thread limit: increase cgroup pids.max (K8s: pid limits / --pod-max-pids) or ulimit -u, then retry.
2. Reduce thread pressure: fewer tokio worker threads per runtime, fewer nested runtimes in tests, fewer concurrent watchdog installs.
3. Free memory / reduce thread stack size (std::thread::Builder supports stack_size; here the crate does not expose it, so address the memory pressure instead).
4. If you control the crate, change install() to log-and-degrade instead of expect so a failed watchdog disables monitoring rather than killing the process.

Example fix

// before (crate code, core/tokio_watchdog/src/lib.rs)
std::thread::Builder::new()
    .name(format!("tokio watchdog {runtime_name}"))
    .spawn(move || { /* ... */ })
    .expect("start watchdog thread");

// after: degrade gracefully instead of panicking
match std::thread::Builder::new()
    .name(format!("tokio watchdog {runtime_name}"))
    .spawn(move || { /* ... */ })
{
    Ok(_handle) => {}
    Err(e) => {
        tracing::warn!(error = %e, runtime = runtime_name, "failed to start watchdog thread; continuing without watchdog");
    }
}

// Best-effort pre-check: ensure we can still spawn a thread before install().
// (There is no portable 'can I spawn' query; a cheap probe is a spawn+join.)
fn can_spawn_thread() -> bool {
    std::thread::Builder::new()
        .spawn(|| {})
        .map(|h| { let _ = h.join(); true })
        .unwrap_or(false)
}

if can_spawn_thread() {
    WatchdogConfig::new(&handle, &registry).install();
} else {
    eprintln!("skipping tokio watchdog: cannot spawn threads (pids/nproc limit?)");
}

When it happens

Trigger: Calling WatchdogConfig::install() on a host already at its thread/process limit (many runtimes x many tokio worker threads plus one watchdog each), inside a container with a low pids.max cgroup limit, under severe memory pressure where the thread stack cannot be mapped, or very late in process shutdown while threads are being torn down.

Common situations: Kubernetes/container deployments with a low pids limit and a thread-heavy InfluxDB 3 process; a hard ulimit -u in CI or on a shared build host; spawning many nested tokio runtimes in tests, each installing its own watchdog; forked child processes inheriting a near-limit thread count.

Related errors

• num_columns_in_parallel should be above zero
• sharder mapped input to non-existant bucket
• mapped to out-of-bounds shard
• If you call `drop_last_value`, the tag buffer must contain a
• If we can remove a value from the interned strings, we must

AI-assisted analysis of influxdata/influxdb@d28e26e048 (2026-08-16). Data as JSON: /api/errors/aa34d2240ff2cd5e. Report an issue: GitHub.