{"record":{"id":"f276240c84e78189","repo":"apache/pulsar","slug":"deadlocked-threads-detected-threadnames","errorCode":null,"errorMessage":"Deadlocked threads detected. ${threadNames}","messagePattern":"Deadlocked threads detected\\. (.+?)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":500,"severity":"critical","filePath":"pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java","lineNumber":474,"sourceCode":"    private void checkDeadlockedThreads() {\n        ThreadMXBean threadBean = ManagementFactory.getThreadMXBean();\n        long[] threadIds = threadBean.findDeadlockedThreads();\n        if (threadIds != null && threadIds.length > 0) {\n            ThreadInfo[] threadInfos = threadBean.getThreadInfo(threadIds, false, false);\n            String threadNames = Arrays.stream(threadInfos)\n                    .map(threadInfo -> threadInfo.getThreadName() + \"(tid=\" + threadInfo.getThreadId() + \")\").collect(\n                            Collectors.joining(\", \"));\n            if (System.currentTimeMillis() - threadDumpLoggedTimestamp\n                    > LOG_THREADDUMP_INTERVAL_WHEN_DEADLOCK_DETECTED) {\n                threadDumpLoggedTimestamp = System.currentTimeMillis();\n                log.error()\n                        .attr(\"detected\", threadNames)\n                        .attr(\"n\", ThreadDumpUtil.buildThreadDiagnosticString())\n                        .log(\"Deadlocked threads detected. \\n\");\n            } else {\n                log.error().attr(\"detected\", threadNames).log(\"Deadlocked threads detected.\");\n            }\n            throw new IllegalStateException(\"Deadlocked threads detected. \" + threadNames);\n        }\n    }\n\n    private CompletableFuture<Void> internalRunHealthCheck() {\n        return pulsar().runHealthCheck(clientAppId());\n    }\n\n    private CompletableFuture<Void> internalDeleteDynamicConfigurationOnMetadataAsync(String configName) {\n        if (!pulsar().getBrokerService().isDynamicConfiguration(configName)) {\n            return FutureUtil.failedFuture(\n                    new RestException(Status.PRECONDITION_FAILED, \"Can't delete non-dynamic configuration\"));\n        } else {\n            return dynamicConfigurationResources().setDynamicConfigurationAsync(old -> {\n                if (old != null) {\n                    old.remove(configName);\n                }\n                return old;\n            });","sourceCodeStart":456,"sourceCodeEnd":492,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java#L456-L492","documentation":"The broker health check (GET /admin/v2/brokers/health) inspects JVM thread dumps for threads in the java.lang.Thread.State BLOCKED deadlock detection (ThreadMXBean.findDeadlockedThreads). If any deadlocked threads are found, it logs a diagnostic and throws IllegalStateException, failing the health check so orchestrators mark the broker unhealthy.","triggerScenarios":"Calling GET /admin/v2/brokers/health while the broker JVM has mutually-cyclically-locked threads (monitor or java.util.concurrent ownable locks), as detected by ThreadMXBean.findDeadlockedThreads.","commonSituations":"BookKeeper/client connection storms causing lock cycles; misuse of synchronized blocks around async callbacks in custom plugins/interceptors; known broker concurrency bugs in specific versions; resource exhaustion (memory pressure, GC pauses) contributing to lock contention; Kubernetes liveness probes repeatedly failing the pod.","solutions":["Capture the thread names and the full dump the error references (ThreadDumpUtil.buildThreadDiagnosticString / jstack on the broker PID) and identify the two or more threads holding cyclic locks.","Check which components own the locks (broker internal code vs. custom plugin) and upgrade Pulsar to the latest patch release where related deadlock bugs may be fixed.","Remove or fix custom code (interceptors, functions, protocol handlers) that acquires multiple locks or blocks inside event-loop callbacks.","If health checks keep failing due to deadlocked background threads, restart the broker to clear the deadlock and file an issue with the thread dump."],"exampleFix":"// before: blocking inside async processing can deadlock lock cycles\nsynchronized (stateLock) {\n    future.get(); // blocks holding lock\n}\n// after: never block event-loop threads; compose futures instead\nreturn CompletableFuture.supplyAsync(() -> computeWithLock(), executor)\n        .thenCompose(this::processAsync);","handlingStrategy":"retry","validationCode":"// pre-check before calling health endpoint\nboolean deadlocked = ((ThreadMXBean) ManagementFactory.getThreadMXBean())\n        .findDeadlockedThreads() != null;\nif (deadlocked) { alert(); }","typeGuard":null,"tryCatchPattern":"try {\n    admin.brokers().healthcheck();\n} catch (PulsarAdminException e) {\n    // message begins with 'Deadlocked threads detected.'\n    if (e.getMessage() != null && e.getMessage().startsWith(\"Deadlocked threads detected\")) {\n        captureThreadDumpAndAlert(e);\n    }\n}","preventionTips":["Avoid blocking calls and multi-lock acquisition on event-loop threads; compose futures instead.","Audit custom plugins/interceptors for synchronized blocks around async operations.","Monitor health-check failures and keep thread-dump capture enabled for post-mortems.","Keep the broker patched to the latest release to benefit from concurrency bug fixes."],"tags":["pulsar","deadlock","jvm","health-check","threads"],"backgroundTag":"deadlocked-threads-detected","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}