deeplearning4j/deeplearning4j · error · RuntimeException

Allocation failed: [" + em + "] for amount of memory " +…

Error message

Allocation failed: [" + em + "] for amount of memory " + numElements * dataType.width() + " bytes

What it means

Thrown by OpaqueDataBuffer.allocateDataBuffer after exhausting MAX_TRIES allocation attempts. Each failed attempt records a memory error (em); native memory allocation could not be satisfied for numElements * dataType.width() bytes even after GC-triggering retries.

Solutions

  1. Reduce the request: smaller batch/shape, or use a narrower dtype (FLOAT16/INT8 where valid).
  2. Increase available native memory: container limits, or set -Dorg.bytedeco.javacpp.maxBytes/-Dorg.bytedeco.javacpp.maxPhysicalBytes.
  3. Enable ND4J workspaces (Nd4j.getWorkspaceManager) to reuse memory instead of repeated large allocations.
  4. Read `em` in the message to identify the specific native allocation error (often OOM).
  5. Call System.gc()/release workspaces before very large allocations; restart long-running services periodically.

Example fix

// before
INDArray w = Nd4j.create(new int[]{500000, 500000}); // ~1TB
// after
long n = 500000L * 500000L; // check feasibility first
if (n * Nd4j.dataType().width() > maxBudgetBytes) {
    throw new IllegalArgumentException("Requested " + n * 4 + " bytes exceeds budget");
}
INDArray w = Nd4j.create(new int[]{50000, 50000});
Defensive patterns

Strategy: validation

Validate before calling

long bytes = numElements * dataType.width();
long maxBytes = Runtime.getRuntime().maxMemory(); // plus configured off-heap budget
if (bytes > maxBudgetBytes) {
    throw new IllegalArgumentException("Requested " + bytes + " bytes exceeds budget " + maxBudgetBytes);
}
INDArray a = Nd4j.create(new long[]{numElements}, dataType);

Try / catch

try {
    INDArray a = Nd4j.create(numElements, dataType);
} catch (RuntimeException e) {
    if (e.getMessage() != null && e.getMessage().startsWith("Allocation failed")) {
        System.gc();
        Nd4j.getWorkspaceManager().destroyAllWorkspacesForCurrentThread();
        // retry once with a smaller request or abort
    }
    throw e;
}

Prevention

When it happens

Trigger: Nd4j.create / buffer allocation requests larger than available off-heap/native memory; repeated failures across MAX_TRIES attempts with the last error code in `em`; requests exceeding configured ND4J memory limits.

Common situations: Oversized tensors (huge batch sizes, embedding tables); container memory limits (Docker/K8s) below ND4J's requested native memory; fragmented native heap in long-running services; missing `-Dorg.bytedeco.javacpp.maxBytes` tuning.

Related errors


AI-assisted analysis of deeplearning4j/deeplearning4j@4c22ac5fe4 (2026-09-07). Data as JSON: /api/errors/7fe98a6657caa043. Report an issue: GitHub.

Appendix: source

Thrown at nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/nativeblas/OpaqueDataBuffer.java:239

                    em = Nd4j.getNativeOps().lastErrorMessage();

                    // if allocation failed it might be caused by casual OOM, so we'll try GC
                    System.gc();

                    // sleeping for 50ms
                    Thread.sleep(50);
                } else {
                    // Buffer is null but no error - shouldn't happen, but break to avoid infinite loop
                    break;
                }
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
                throw new RuntimeException("Allocation interrupted", e);
            }
        }

        // if MAX_TRIES is over, we'll just throw an exception
        throw new RuntimeException("Allocation failed: [" + em + "] for amount of memory " + numElements * dataType.width() + " bytes");
    }

    /**
     * This method expands buffer, and copies content to the new buffer
     *
     * PLEASE NOTE: if InteropDataBuffer doesn't own actual buffers - original pointers won't be released
     * @param numElements
     */
    public void expand(long numElements) {
        int ec = 0;
        String em = null;

        for (int t = 0; t < MAX_TRIES; t++) {
            try {
                // try to expand the buffer
                Nd4j.getNativeOps().dbExpand(this, numElements);

                // check error code

View on GitHub (pinned to 4c22ac5fe4)