{"record":{"id":"5bafc02a74d0d945","repo":"elastic/elasticsearch","slug":"no-gpu-resources-available-and-unable-to-create-ne","errorCode":null,"errorMessage":"No GPU resources available and unable to create new ones","messagePattern":"No GPU resources available and unable to create new ones","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/CuVSResourceManager.java","lineNumber":236,"sourceCode":"                    if (res != null) {\n                        // Check immutable constraints\n                        long totalMemoryInBytes = gpuMemoryService.totalMemoryInBytes(res);\n                        long availableMemoryInBytes = gpuMemoryService.availableMemoryInBytes(res);\n                        enoughMemory = requiredMemoryInBytes <= availableMemoryInBytes;\n                        logMemoryCheck(availableMemoryInBytes, totalMemoryInBytes, requiredMemoryInBytes, enoughMemory);\n\n                        if (requiredMemoryInBytes > totalMemoryInBytes) {\n                            throw memoryExceededError(numVectors, dims, totalMemoryInBytes);\n                        }\n\n                        // If no resource in the pool is locked, we must proceed to avoid livelock\n                        if (enoughMemory == false && numLockedResources() == 0) {\n                            logLivelockBypass(availableMemoryInBytes, requiredMemoryInBytes);\n                            break;\n                        }\n                    } else {\n                        if (nonBlocking == false && createdCount == 0) {\n                            throw new IOException(\"No GPU resources available and unable to create new ones\");\n                        }\n                        logger.debug(\"No resources available in pool\");\n                        enoughMemory = false;\n                    }\n\n                    allConditionsMet = enoughMemory;\n                    if (allConditionsMet == false) {\n                        if (nonBlocking) {\n                            return null;\n                        }\n                        logger.debug(\"Waiting for GPU resources for [{}]\", reason);\n                        enoughResourcesCondition.await();\n                    }\n                }\n                logAcquired(reason, started, requiredMemoryInBytes);\n                gpuMemoryService.reserveMemory(requiredMemoryInBytes);\n                res.lock(() -> gpuMemoryService.releaseMemory(requiredMemoryInBytes));\n                return res;","sourceCodeStart":218,"sourceCodeEnd":254,"githubUrl":"https://github.com/elastic/elasticsearch/blob/db6a809a667c081ca1dc7500389d26975573215f/libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/CuVSResourceManager.java#L218-L254","documentation":"Thrown (as IOException) inside PoolingCuVSResourceManager.acquireResource when the pool has no free resource, createdCount is 0 (nothing has ever been created), and the call is blocking (nonBlocking == false). This means no GPU resource can be returned and none can be allocated — a hard unavailability state rather than a transient wait.","triggerScenarios":"A blocking acquireResource call when the pool is empty and no ManagedCuVSResources has been created (createdCount == 0). This occurs when GPU initialization failed or was skipped so the pool never populated, and a writer/reader then requests a resource.","commonSituations":"GPU support is enabled in config but no compatible GPU is present on the node (initializeGpuInfo returned UNSUPPORTED); the pool was constructed but never warmed; a race during node startup where a request arrives before resources are provisioned; GPU driver/CUDA runtime failure preventing resource creation.","solutions":["Verify a compatible GPU is present and the driver/CUDA runtime is healthy (check CuVSGPUSupport.instance() reports a valid GpuInfo, not UNSUPPORTED).","Ensure the resource pool is initialized/warmed before serving requests, or gate GPU codec usage on GPU availability.","If GPU is optional, fall back to a CPU codec path when this IOException is caught, or disable the GPU codec feature.","Use nonBlocking acquire (returns null instead of throwing) and handle null gracefully."],"exampleFix":"// before: blocking acquire with no GPU available\nManagedCuVSResources res = manager.acquireResource(n, d, dt, params, false, reason);\n\n// after: check GPU support first, fall back if unavailable\nif (CuVSGPUSupport.instance().isGpuSupported() == false) {\n    throw new IllegalStateException(\"GPU codec requested but no GPU available\");\n}\nManagedCuVSResources res = manager.acquireResource(n, d, dt, params, false, reason);","handlingStrategy":"try-catch","validationCode":"static void assertGpuAvailable() {\n    if (!CuVSGPUSupport.instance().isGpuSupported()) {\n        throw new IllegalStateException(\"no compatible GPU available for GPU codec\");\n    }\n}","typeGuard":null,"tryCatchPattern":"try {\n    ManagedCuVSResources res = manager.acquireResource(n, d, dt, params, false, reason);\n} catch (IOException e) {\n    if (e.getMessage().equals(\"No GPU resources available and unable to create new ones\")) {\n        // fall back to CPU codec path or fail the shard operation gracefully\n    }\n    throw e;\n}","preventionTips":["Verify GPU availability via CuVSGPUSupport before enabling the GPU codec.","Warm the resource pool during node startup.","Consider nonBlocking acquire and handle null gracefully.","Have a CPU codec fallback path documented and ready."],"tags":["gpu","resource-pool","elasticsearch","codec","io","availability"],"backgroundTag":null,"analyzedSha":"db6a809a667c081ca1dc7500389d26975573215f","analyzedAt":"2026-08-12T01:39:14.192Z","schemaVersion":2},"datasetVersion":"2026-08-12T13:17:24.610Z"}