{"record":{"id":"aa34d2240ff2cd5e","repo":"influxdata/influxdb","slug":"start-watchdog-thread","errorCode":null,"errorMessage":"start watchdog thread","messagePattern":"start watchdog thread","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"core/tokio_watchdog/src/lib.rs","lineNumber":181,"sourceCode":"                            let Some(d) = rx_response.blocking_recv() else {\n                                return;\n                            };\n                            debug!(\n                                runtime = runtime_name,\n                                hang_secs = d.as_secs_f64(),\n                                \"tokio stops hanging\",\n                            );\n                            d\n                        }\n                        Err(TryRecvError::Disconnected) => {\n                            return;\n                        }\n                    };\n\n                    metric_latency.record(d);\n                }\n            })\n            .expect(\"start watchdog thread\");\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::sync::Arc;\n\n    use test_helpers::tracing::TracingCapture;\n\n    use super::*;\n\n    #[tokio::test]\n    #[should_panic(expected = \"sum of tick and warn duration must be non-zero\")]\n    async fn test_panic_zero_duration() {\n        let registry = Registry::default();\n        WatchdogConfig::new(&Handle::current(), &registry)\n            .with_tick_duration(Duration::ZERO)\n            .with_warn_duration(Duration::ZERO)","sourceCodeStart":163,"sourceCodeEnd":199,"githubUrl":"https://github.com/influxdata/influxdb/blob/d28e26e048401c53cbb98cf2d6ab0cf1e98048ca/core/tokio_watchdog/src/lib.rs#L163-L199","documentation":"WatchdogConfig::install() spawns a dedicated OS thread ('tokio watchdog <name>') that periodically pings the tokio runtime to detect hangs. The .expect('start watchdog thread') panics when std::thread::Builder::spawn returns Err, i.e. the OS refused to create the thread. This is an environment/resource failure, not a logic bug: the thread limit (RLIMIT_NPROC / cgroup pids.max), memory exhaustion, or a forked child context typically prevents the spawn.","triggerScenarios":"Calling WatchdogConfig::install() on a host already at its thread/process limit (many runtimes x many tokio worker threads plus one watchdog each), inside a container with a low pids.max cgroup limit, under severe memory pressure where the thread stack cannot be mapped, or very late in process shutdown while threads are being torn down.","commonSituations":"Kubernetes/container deployments with a low pids limit and a thread-heavy InfluxDB 3 process; a hard ulimit -u in CI or on a shared build host; spawning many nested tokio runtimes in tests, each installing its own watchdog; forked child processes inheriting a near-limit thread count.","solutions":["Raise the process/thread limit: increase cgroup pids.max (K8s: pid limits / --pod-max-pids) or ulimit -u, then retry.","Reduce thread pressure: fewer tokio worker threads per runtime, fewer nested runtimes in tests, fewer concurrent watchdog installs.","Free memory / reduce thread stack size (std::thread::Builder supports stack_size; here the crate does not expose it, so address the memory pressure instead).","If you control the crate, change install() to log-and-degrade instead of expect so a failed watchdog disables monitoring rather than killing the process."],"exampleFix":"// before (crate code, core/tokio_watchdog/src/lib.rs)\nstd::thread::Builder::new()\n    .name(format!(\"tokio watchdog {runtime_name}\"))\n    .spawn(move || { /* ... */ })\n    .expect(\"start watchdog thread\");\n\n// after: degrade gracefully instead of panicking\nmatch std::thread::Builder::new()\n    .name(format!(\"tokio watchdog {runtime_name}\"))\n    .spawn(move || { /* ... */ })\n{\n    Ok(_handle) => {}\n    Err(e) => {\n        tracing::warn!(error = %e, runtime = runtime_name, \"failed to start watchdog thread; continuing without watchdog\");\n    }\n}","handlingStrategy":"fallback","validationCode":"// Best-effort pre-check: ensure we can still spawn a thread before install().\n// (There is no portable 'can I spawn' query; a cheap probe is a spawn+join.)\nfn can_spawn_thread() -> bool {\n    std::thread::Builder::new()\n        .spawn(|| {})\n        .map(|h| { let _ = h.join(); true })\n        .unwrap_or(false)\n}\n\nif can_spawn_thread() {\n    WatchdogConfig::new(&handle, &registry).install();\n} else {\n    eprintln!(\"skipping tokio watchdog: cannot spawn threads (pids/nproc limit?)\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Raise container/cgroup pid limits (K8s pid limits, --pids-limit) and ulimit -u for thread-heavy processes.","Reduce concurrent tokio runtimes in tests; each install() adds a real OS thread.","If embedding tokio_watchdog, prefer a degrade-not-die spawn wrapper (log a warning on spawn failure) over expect."],"tags":["rust","tokio","watchdog","thread-limit","resources","panic","influxdb"],"backgroundTag":"thread-spawn-failure","analyzedSha":"d28e26e048401c53cbb98cf2d6ab0cf1e98048ca","analyzedAt":"2026-08-16T19:53:34.623Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}