deeplearning4j/deeplearning4j · error · RuntimeException
Allocation failed: [" + em + "] for amount of memory " +…
Error message
Allocation failed: [" + em + "] for amount of memory " + numElements * dataType.width() + " bytes
What it means
Thrown by OpaqueDataBuffer.allocateDataBuffer after exhausting MAX_TRIES allocation attempts. Each failed attempt records a memory error (em); native memory allocation could not be satisfied for numElements * dataType.width() bytes even after GC-triggering retries.
Solutions
- Reduce the request: smaller batch/shape, or use a narrower dtype (FLOAT16/INT8 where valid).
- Increase available native memory: container limits, or set -Dorg.bytedeco.javacpp.maxBytes/-Dorg.bytedeco.javacpp.maxPhysicalBytes.
- Enable ND4J workspaces (Nd4j.getWorkspaceManager) to reuse memory instead of repeated large allocations.
- Read `em` in the message to identify the specific native allocation error (often OOM).
- Call System.gc()/release workspaces before very large allocations; restart long-running services periodically.
Example fix
// before
INDArray w = Nd4j.create(new int[]{500000, 500000}); // ~1TB
// after
long n = 500000L * 500000L; // check feasibility first
if (n * Nd4j.dataType().width() > maxBudgetBytes) {
throw new IllegalArgumentException("Requested " + n * 4 + " bytes exceeds budget");
}
INDArray w = Nd4j.create(new int[]{50000, 50000}); Defensive patterns
Strategy: validation
Validate before calling
long bytes = numElements * dataType.width();
long maxBytes = Runtime.getRuntime().maxMemory(); // plus configured off-heap budget
if (bytes > maxBudgetBytes) {
throw new IllegalArgumentException("Requested " + bytes + " bytes exceeds budget " + maxBudgetBytes);
}
INDArray a = Nd4j.create(new long[]{numElements}, dataType); Try / catch
try {
INDArray a = Nd4j.create(numElements, dataType);
} catch (RuntimeException e) {
if (e.getMessage() != null && e.getMessage().startsWith("Allocation failed")) {
System.gc();
Nd4j.getWorkspaceManager().destroyAllWorkspacesForCurrentThread();
// retry once with a smaller request or abort
}
throw e;
} Prevention
- Compute bytes = numElements * dtype.width() and check against a memory budget before allocating.
- Set -Dorg.bytedeco.javacpp.maxBytes / maxPhysicalBytes to match your container limits.
- Use ND4J workspaces for repeated large allocations.
- Parse `em` from the message to confirm OOM vs other native errors.
- Keep batch sizes and shapes within tested bounds.
When it happens
Trigger: Nd4j.create / buffer allocation requests larger than available off-heap/native memory; repeated failures across MAX_TRIES attempts with the last error code in `em`; requests exceeding configured ND4J memory limits.
Common situations: Oversized tensors (huge batch sizes, embedding tables); container memory limits (Docker/K8s) below ND4J's requested native memory; fragmented native heap in long-running services; missing `-Dorg.bytedeco.javacpp.maxBytes` tuning.
Related errors
- DataBuffer expansion failed: [" + em + "]
- DataBuffer view creation failed: [" + em + "]
- Allocation interrupted
- Failed to register buffer with DeallocatorService
- A columns must equal B rows. MMul attempt:
AI-assisted analysis of deeplearning4j/deeplearning4j@4c22ac5fe4 (2026-09-07).
Data as JSON: /api/errors/7fe98a6657caa043.
Report an issue: GitHub.
Appendix: source
Thrown at nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/nativeblas/OpaqueDataBuffer.java:239
em = Nd4j.getNativeOps().lastErrorMessage();
// if allocation failed it might be caused by casual OOM, so we'll try GC
System.gc();
// sleeping for 50ms
Thread.sleep(50);
} else {
// Buffer is null but no error - shouldn't happen, but break to avoid infinite loop
break;
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException("Allocation interrupted", e);
}
}
// if MAX_TRIES is over, we'll just throw an exception
throw new RuntimeException("Allocation failed: [" + em + "] for amount of memory " + numElements * dataType.width() + " bytes");
}
/**
* This method expands buffer, and copies content to the new buffer
*
* PLEASE NOTE: if InteropDataBuffer doesn't own actual buffers - original pointers won't be released
* @param numElements
*/
public void expand(long numElements) {
int ec = 0;
String em = null;
for (int t = 0; t < MAX_TRIES; t++) {
try {
// try to expand the buffer
Nd4j.getNativeOps().dbExpand(this, numElements);
// check error codeView on GitHub (pinned to 4c22ac5fe4)