openjdk/jdk · critical

java.lang.OutOfMemoryError\n

Error message

java.lang.OutOfMemoryError\n

What it means

Windows AWT installs a C++ new-handler; when malloc/operator new exhausts native heap, NewHandler::handler prints 'java.lang.OutOfMemoryError' to stderr and returns FALSE so the allocation fails immediately (subsequent safe_Malloc/safe_Realloc/safe_Calloc then throw std::bad_alloc). The line means the process ran out of native (C heap) memory inside AWT code, not a Java heap OOM — despite the java-looking text.

Source

Thrown at src/java.desktop/windows/native/libawt/windows/awt_new.cpp:77

    DASSERT(thread_seeded != TLS_OUT_OF_INDEXES);
#endif

    // use new handler for operator new and malloc
    _set_new_mode(1);

    // set the function which will be called when operator new or
    // malloc runs out of memory
    _set_new_handler((_PNH)NewHandler::handler);
}

// Called when malloc or operator new runs out of memory. We try to
// compact the heap by initiating a Java GC. If the amount of free
// memory available after this operation increases, then we return
// (1) to indicate that malloc or operator new should retry the
// allocation. Returning (0) indicates that the allocation should fail.
int
NewHandler::handler(size_t) {
    fprintf(stderr, "java.lang.OutOfMemoryError\n");
    return FALSE;
}

// These three functions throw std::bad_alloc in an out of memory condition
// instead of returning 0. safe_Realloc will return 0 if memblock is not
// NULL and size is 0. safe_Malloc and safe_Calloc will never return 0.
void *safe_Malloc(size_t size) {
    void *ptr = malloc(size);
    if (ptr == nullptr) {
        throw std::bad_alloc();
    }

    return ptr;
}

void *safe_Calloc(size_t num, size_t size) {
    void *ptr = calloc(num, size);
    if (ptr == nullptr) {

View on GitHub (pinned to 88dfb74bbe)

Solutions

1. Switch to a 64-bit JVM — removes the address-space ceiling that most often triggers this
2. On 32-bit, lower -Xmx/-XX:MaxPermSize equivalents to leave room for native heap and stacks
3. Profile native memory (jcmd VM.native_memory with NativeMemoryTracking, Windows UMDH) to find C/GDI leaks
4. Reduce simultaneous large images/back buffers; call Image.flush()/dispose paths so GDI/bitmaps are released

Example fix

# before (32-bit JVM, huge heap)
java -Xmx1600m -jar app.jar   # native heap starved -> 'java.lang.OutOfMemoryError' from NewHandler

# after
java -Xmx900m -jar app.jar     # or use a 64-bit JVM and keep -Xmx

// before launch: ensure native heap headroom (32-bit especially)
// keep -Xmx well below the 32-bit process limit, or check bitness:
if (System.getProperty("sun.arch.data.model").equals("32")
        && Runtime.getRuntime().maxMemory() > 1_200_000_000L) {
    System.err.println("32-bit JVM with very large -Xmx: native heap may be starved");
}

try { bigOp(); } catch (OutOfMemoryError | std-bad-alloc-wrapper e) { /* reduce image/buffer sizes, flush caches, retry smaller */ }

When it happens

Trigger: Heavy native allocation on Windows: very large images/bitmaps, many DWM/GDI objects, deep component trees, native buffers in 2D/3D pipelines. malloc cannot be satisfied, the handler runs (the comment says it tries to trigger a Java GC on some paths, but here it simply reports and fails), and std::bad_alloc propagates from safe_* allocators, often crashing or exiting the app.

Common situations: 32-bit JVM hitting the ~2GB process address-space ceiling with big heaps (-Xmx too large leaves too little native space); memory leaks in native code (GDI/bitmaps not released); loading huge images; long-running desktop apps fragmenting the C heap.

Related errors

• Failed to delete file.
• Error: Out of memory in ADLC\n
• failed to get the full path of the executable: %lu
• no '\' found in the full path of the executable
• CreateProcess failed: %lu

AI-assisted analysis of openjdk/jdk@88dfb74bbe (2026-08-14). Data as JSON: /api/errors/a3fd81bf0726cd8c. Report an issue: GitHub.