apache/pulsar · critical · IllegalStateException

Deadlocked threads detected. ${threadNames}

Error message

Deadlocked threads detected. ${threadNames}

What it means

The broker health check (GET /admin/v2/brokers/health) inspects JVM thread dumps for threads in the java.lang.Thread.State BLOCKED deadlock detection (ThreadMXBean.findDeadlockedThreads). If any deadlocked threads are found, it logs a diagnostic and throws IllegalStateException, failing the health check so orchestrators mark the broker unhealthy.

Source

Thrown at pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java:474

    private void checkDeadlockedThreads() {
        ThreadMXBean threadBean = ManagementFactory.getThreadMXBean();
        long[] threadIds = threadBean.findDeadlockedThreads();
        if (threadIds != null && threadIds.length > 0) {
            ThreadInfo[] threadInfos = threadBean.getThreadInfo(threadIds, false, false);
            String threadNames = Arrays.stream(threadInfos)
                    .map(threadInfo -> threadInfo.getThreadName() + "(tid=" + threadInfo.getThreadId() + ")").collect(
                            Collectors.joining(", "));
            if (System.currentTimeMillis() - threadDumpLoggedTimestamp
                    > LOG_THREADDUMP_INTERVAL_WHEN_DEADLOCK_DETECTED) {
                threadDumpLoggedTimestamp = System.currentTimeMillis();
                log.error()
                        .attr("detected", threadNames)
                        .attr("n", ThreadDumpUtil.buildThreadDiagnosticString())
                        .log("Deadlocked threads detected. \n");
            } else {
                log.error().attr("detected", threadNames).log("Deadlocked threads detected.");
            }
            throw new IllegalStateException("Deadlocked threads detected. " + threadNames);
        }
    }

    private CompletableFuture<Void> internalRunHealthCheck() {
        return pulsar().runHealthCheck(clientAppId());
    }

    private CompletableFuture<Void> internalDeleteDynamicConfigurationOnMetadataAsync(String configName) {
        if (!pulsar().getBrokerService().isDynamicConfiguration(configName)) {
            return FutureUtil.failedFuture(
                    new RestException(Status.PRECONDITION_FAILED, "Can't delete non-dynamic configuration"));
        } else {
            return dynamicConfigurationResources().setDynamicConfigurationAsync(old -> {
                if (old != null) {
                    old.remove(configName);
                }
                return old;
            });

View on GitHub (pinned to 820761864e)

Solutions

1. Capture the thread names and the full dump the error references (ThreadDumpUtil.buildThreadDiagnosticString / jstack on the broker PID) and identify the two or more threads holding cyclic locks.
2. Check which components own the locks (broker internal code vs. custom plugin) and upgrade Pulsar to the latest patch release where related deadlock bugs may be fixed.
3. Remove or fix custom code (interceptors, functions, protocol handlers) that acquires multiple locks or blocks inside event-loop callbacks.
4. If health checks keep failing due to deadlocked background threads, restart the broker to clear the deadlock and file an issue with the thread dump.

Example fix

// before: blocking inside async processing can deadlock lock cycles
synchronized (stateLock) {
    future.get(); // blocks holding lock
}
// after: never block event-loop threads; compose futures instead
return CompletableFuture.supplyAsync(() -> computeWithLock(), executor)
        .thenCompose(this::processAsync);

// pre-check before calling health endpoint
boolean deadlocked = ((ThreadMXBean) ManagementFactory.getThreadMXBean())
        .findDeadlockedThreads() != null;
if (deadlocked) { alert(); }

try {
    admin.brokers().healthcheck();
} catch (PulsarAdminException e) {
    // message begins with 'Deadlocked threads detected.'
    if (e.getMessage() != null && e.getMessage().startsWith("Deadlocked threads detected")) {
        captureThreadDumpAndAlert(e);
    }
}

When it happens

Trigger: Calling GET /admin/v2/brokers/health while the broker JVM has mutually-cyclically-locked threads (monitor or java.util.concurrent ownable locks), as detected by ThreadMXBean.findDeadlockedThreads.

Common situations: BookKeeper/client connection storms causing lock cycles; misuse of synchronized blocks around async callbacks in custom plugins/interceptors; known broker concurrency bugs in specific versions; resource exhaustion (memory pressure, GC pauses) contributing to lock contention; Kubernetes liveness probes repeatedly failing the pod.

Related errors

• The topic has a max partition index of %d, the number of par
• entryFilterNames can't be empty. To remove entry filters use
• The offloadPolicies must be specified for namespace offload.
• The driver is not supported, support value: ${supportedDrive
• The bucket must be specified for namespace offload.

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/f276240c84e78189. Report an issue: GitHub.