apache/seatunnel · error · NoEnoughResourceException

can't apply resource request with retry times

Error message

can't apply resource request with retry times: ${MAX_RETRY_TIMES}

What it means

ResourceRequestHandler retries slot requests up to MAX_RETRY_TIMES when the cluster lacks enough resources. If requestSlotWithRetry is invoked with a retryTimes counter that has already been exhausted (<=0), it immediately completes a failed future with NoEnoughResourceException instead of asking workers for slots. This means the resource manager gave up after all retries without securing the requested ResourceProfiles.

Solutions

1. Free up cluster resources: stop/finish other running jobs or scale up worker nodes (increase worker CPU/memory/slot counts)
2. Check job resource settings in the config (resource tags, cpu/memory per pipeline) and reduce them to fit available capacity
3. Verify MAX_RETRY_TIMES and increase it (engine config) so transient resource contention can be retried successfully
4. Inspect worker logs via REST /hazelcast/rest/map/running-job or cluster metrics to confirm which workers failed the pre-check

Example fix

// before: request larger resources than cluster has
resource { cpu = 16; memory = "64G" }
// after: match worker capacity
resource { cpu = 2; memory = "4G" }

// compare requested vs available via REST
curl http://master:8080/hazelcast/rest/maps/used-resource  # ensure capacity >= request

try {
    slotFuture.get();
} catch (ExecutionException e) {
    if (e.getCause() instanceof NoEnoughResourceException) {
        // backoff, free/reduce resources, then retry
    }
}

When it happens

Trigger: Calling ResourceManager.request()/requestSlotWithRetry when retry budget is spent, i.e. the job could not get slots after MAX_RETRY_TIMES rounds of preCheck+request against all workers.

Common situations: Cluster has fewer available CPU/memory on worker nodes than the job's resource requirements; too many concurrent jobs exhausting slots; request tags matching no worker; misconfigured slot-num or dynamic resource profiles.

Related errors

• can't apply resource request
• Can not get coordinator service from an active master node.
• checkpoint restore requires a terminal source job…
• close event processor error
• cluster have no master node

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/04ddbd77908b8e16. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/resourcemanager/ResourceRequestHandler.java:105

        this.registerWorker = registerWorker;
        this.resourceManager = resourceManager;
        this.allocateStrategy =
                resourceManager.getEngineConfig().getSlotServiceConfig().getAllocateStrategy();
        this.slotAllocationStrategy = slotAllocationStrategy;
    }

    public CompletableFuture<List<SlotProfile>> request(Map<String, String> tags) {
        requestSlotWithRetry(resourceProfile, MAX_RETRY_TIMES, tags);
        return completableFuture;
    }

    private CompletableFuture<SlotAndWorkerProfile> requestSlotWithRetry(
            List<ResourceProfile> request, int retryTimes, Map<String, String> tags) {
        if (retryTimes <= 0) {
            LOGGER.fine("can't apply resource request with retry times: " + MAX_RETRY_TIMES);
            return CompletableFuture.supplyAsync(
                    () -> {
                        throw new NoEnoughResourceException(
                                "can't apply resource request with retry times: "
                                        + MAX_RETRY_TIMES);
                    });
        }
        List<CompletableFuture<SlotAndWorkerProfile>> allRequestFuture = requestSlots(request);
        // all resource preCheck done, also had sent request to worker
        return getAllOfFuture(allRequestFuture)
                .whenComplete(
                        withTryCatch(
                                LOGGER,
                                (unused, error) -> {
                                    if (error != null) {
                                        completeRequestWithException(toExternalException(error));
                                    } else {
                                        List<ResourceProfile> needRequestResource =
                                                stillNeedRequestResource();
                                        if (!needRequestResource.isEmpty()) {
                                            Exception requestSlotWithRetryError = null;

View on GitHub (pinned to cf67b549a7)