elastic/elasticsearch · critical · IOException

No GPU resources available and unable to create new ones

Error message

No GPU resources available and unable to create new ones

What it means

Thrown (as IOException) inside PoolingCuVSResourceManager.acquireResource when the pool has no free resource, createdCount is 0 (nothing has ever been created), and the call is blocking (nonBlocking == false). This means no GPU resource can be returned and none can be allocated — a hard unavailability state rather than a transient wait.

Source

Thrown at libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/CuVSResourceManager.java:236

                    if (res != null) {
                        // Check immutable constraints
                        long totalMemoryInBytes = gpuMemoryService.totalMemoryInBytes(res);
                        long availableMemoryInBytes = gpuMemoryService.availableMemoryInBytes(res);
                        enoughMemory = requiredMemoryInBytes <= availableMemoryInBytes;
                        logMemoryCheck(availableMemoryInBytes, totalMemoryInBytes, requiredMemoryInBytes, enoughMemory);

                        if (requiredMemoryInBytes > totalMemoryInBytes) {
                            throw memoryExceededError(numVectors, dims, totalMemoryInBytes);
                        }

                        // If no resource in the pool is locked, we must proceed to avoid livelock
                        if (enoughMemory == false && numLockedResources() == 0) {
                            logLivelockBypass(availableMemoryInBytes, requiredMemoryInBytes);
                            break;
                        }
                    } else {
                        if (nonBlocking == false && createdCount == 0) {
                            throw new IOException("No GPU resources available and unable to create new ones");
                        }
                        logger.debug("No resources available in pool");
                        enoughMemory = false;
                    }

                    allConditionsMet = enoughMemory;
                    if (allConditionsMet == false) {
                        if (nonBlocking) {
                            return null;
                        }
                        logger.debug("Waiting for GPU resources for [{}]", reason);
                        enoughResourcesCondition.await();
                    }
                }
                logAcquired(reason, started, requiredMemoryInBytes);
                gpuMemoryService.reserveMemory(requiredMemoryInBytes);
                res.lock(() -> gpuMemoryService.releaseMemory(requiredMemoryInBytes));
                return res;

View on GitHub (pinned to db6a809a66)

Solutions

1. Verify a compatible GPU is present and the driver/CUDA runtime is healthy (check CuVSGPUSupport.instance() reports a valid GpuInfo, not UNSUPPORTED).
2. Ensure the resource pool is initialized/warmed before serving requests, or gate GPU codec usage on GPU availability.
3. If GPU is optional, fall back to a CPU codec path when this IOException is caught, or disable the GPU codec feature.
4. Use nonBlocking acquire (returns null instead of throwing) and handle null gracefully.

Example fix

// before: blocking acquire with no GPU available
ManagedCuVSResources res = manager.acquireResource(n, d, dt, params, false, reason);

// after: check GPU support first, fall back if unavailable
if (CuVSGPUSupport.instance().isGpuSupported() == false) {
    throw new IllegalStateException("GPU codec requested but no GPU available");
}
ManagedCuVSResources res = manager.acquireResource(n, d, dt, params, false, reason);

static void assertGpuAvailable() {
    if (!CuVSGPUSupport.instance().isGpuSupported()) {
        throw new IllegalStateException("no compatible GPU available for GPU codec");
    }
}

try {
    ManagedCuVSResources res = manager.acquireResource(n, d, dt, params, false, reason);
} catch (IOException e) {
    if (e.getMessage().equals("No GPU resources available and unable to create new ones")) {
        // fall back to CPU codec path or fail the shard operation gracefully
    }
    throw e;
}

When it happens

Trigger: A blocking acquireResource call when the pool is empty and no ManagedCuVSResources has been created (createdCount == 0). This occurs when GPU initialization failed or was skipped so the pool never populated, and a writer/reader then requests a resource.

Common situations: GPU support is enabled in config but no compatible GPU is present on the node (initializeGpuInfo returned UNSUPPORTED); the pool was constructed but never warmed; a race during node startup where a request arrives before resources are provisioned; GPU driver/CUDA runtime failure preventing resource creation.

Related errors

• Resource count must be between 1 and 4
• {} must be non-negative, got [{}]
• Dataset dimensions must be positive: rows={}, features={}
• input cannot be null
• PGP exception during signature verification for [{}]

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/5bafc02a74d0d945. Report an issue: GitHub.