apache/pulsar · warning · CompletionException

Failed due to heap memory limit exceeded

Error message

Failed due to heap memory limit exceeded

What it means

A 429 TOO_MANY_REQUESTS thrown when the initial acquisition of in-flight memory permits for a namespace topic-list operation fails the broker's heap memory limit (AsyncDualMemoryLimiter with LimitType.HEAP_MEMORY). The broker bounds total memory used by concurrent topic-list responses; when the estimated list size would exceed the in-flight heap budget, it rejects the request instead of risking OOM.

Source

Thrown at pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java:216

                    // for failed request
                    // handle resetting the TopicListSizeResultCache.ResultHolder
                    listSizeHolder.resetIfInitializing();
                    // cancel any pending permit request
                    permitRequestCancelled.set(true);
                }
                AsyncDualMemoryLimiter.AsyncDualMemoryLimiterPermit initialPermit = initialPermitsRef.get();
                if (initialPermit != null) {
                    maxTopicListInFlightLimiter.release(initialPermit);
                }
                AsyncDualMemoryLimiter.AsyncDualMemoryLimiterPermit permits = permitsRef.get();
                if (permits != null) {
                    maxTopicListInFlightLimiter.release(permits);
                }
            }
        });
        return listSizeHolder.getSizeAsync().thenCompose(initialSize -> maxTopicListInFlightLimiter.acquire(initialSize,
                AsyncDualMemoryLimiter.LimitType.HEAP_MEMORY, isPermitRequestCancelled).exceptionally(t -> {
            throw new CompletionException(
                    new RestException(Status.TOO_MANY_REQUESTS, "Failed due to heap memory limit exceeded"));
        }).thenCompose(initialPermits -> {
            initialPermitsRef.set(initialPermits);
            // perform the actual get list of topics operation
            return doInternalGetListOfTopics(policies, mode).thenCompose(topicList -> {
                long actualSize = TopicListMemoryLimiter.estimateTopicListSize(topicList);
                listSizeHolder.updateSize(actualSize);
                return maxTopicListInFlightLimiter.update(initialPermits, actualSize, isPermitRequestCancelled)
                        .exceptionally(t -> {
                            throw new CompletionException(new RestException(Status.TOO_MANY_REQUESTS,
                                    "Failed due to heap memory limit exceeded"));
                        }).thenApply(permits -> {
                            permitsRef.set(permits);
                            initialPermitsRef.set(null);
                            return topicList;
                        });
            });
        }));

View on GitHub (pinned to 820761864e)

Solutions

1. Reduce the number of concurrent topic-list requests against the broker (add client-side rate limiting/serialization).
2. Use pagination/filtering where available (e.g. getTopics with mode filters) or narrower namespace partitioning to shrink result size.
3. Increase broker heap (-Xmx) and/or the max topic-list in-flight memory limiter configuration to raise the budget.
4. Retry the request after backing off — the limiter is in-flight based, so as other requests complete, permits free up.

Example fix

// before: unbounded parallel listing
namespaces.forEach(ns -> admin.namespaces().getNamespaces(ns));
// after: bounded concurrency + retry on 429
ExecutorService pool = Executors.newFixedThreadPool(2);
// on RestException 429: back off and retry with exponential delay

// client-side: avoid listing very large namespaces concurrently
if (knownTopicCount(ns) > LARGE_NS_THRESHOLD) { /* use pagination / serialize */ }

try { admin.namespaces().getNamespaces(tenant + "/" + ns); }
catch (PulsarAdminException e) {
  if (e.getStatusCode() == 429) { /* back off and retry, or reduce parallelism */ }
  else throw e;
}

When it happens

Trigger: GET /admin/v3/namespaces/{tenant}/{ns}/topics (or ?mode=all/persistent/non-persistent) on a namespace with a very large number of topics while the broker's topic-list in-flight memory budget is already (nearly) exhausted by concurrent list requests.

Common situations: Huge namespaces (tens of thousands of topics) whose list results exceed the limiter's heap threshold; monitoring/automation polling many namespaces' topic lists concurrently; undersized broker heap making the budget small.

Related errors

• Max message size need smaller than jvm directMemory
• brokerDeleteInactiveTopicsEnabled and brokerCloseInactiveTop
• brokerCloseInactiveTopicsEnabled only supports brokerDeleteI
• cluster data is required
• Cluster already exists

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/2e6910d1c415c30f. Report an issue: GitHub.