openjdk/jdk · error

OOM error in native tmp buffer allocation

Error message

OOM error in native tmp buffer allocation

What it means

Windows counterpart of basePath(): it scans for the last path separator and mallocs a copy of the directory portion of a path during libinstrument's agent JAR resolution on Windows. If malloc fails, this message is printed and NULL returned, aborting agent startup. It is a native allocation failure, independent of the path's correctness.

Source

Thrown at src/java.instrument/windows/native/libinstrument/FileSystemSupport_md.c:62

static int isLetter(char c) {
    return ((c >= 'a') && (c <= 'z')) || ((c >= 'A') && (c <= 'Z'));
}

char* basePath(const char* path) {
    char* pos = strchr(path, slash);
    char* last = NULL;
    while (pos != NULL) {
        last = pos;
        pos++;
        pos = strchr(pos, slash);
    }
    if (last == NULL) {
        return (char*)path;
    } else {
        int len = (int)(last - path);
        char* str = (char*)malloc(len+1);
        if (str == NULL) {
            fprintf(stderr, "OOM error in native tmp buffer allocation");
            return NULL;
        }
        if (len > 0) {
            memcpy(str, path, len);
        }
        str[len] = '\0';
        return str;
    }
}



/* -- Normalization - src/windows/classes/java/io/Win32FileSystem.java */


/* A normal Win32 pathname contains no duplicate slashes, except possibly
 * for a UNC prefix, and does not end with a slash.  It may be the empty
 * string.  Normalized Win32 pathnames have the convenient property that

View on GitHub (pinned to 88dfb74bbe)

Solutions

1. Free system memory or raise the commit limit / Job Object cap before starting the JVM
2. If using a 32-bit JVM with a big -Xmx, reduce heap reservation or move to 64-bit so native allocations have room
3. Check for DLL-injecting security software competing for native memory at startup
4. Reproduce outside the constrained environment to confirm the agent and path are fine

Example fix

rem before (32-bit JVM, -Xmx1600m leaves no native headroom)
java -Xmx1600m -javaagent:C:\agents\agent.jar -jar app.jar

rem after
java -Xmx1024m -javaagent:C:\agents\agent.jar -jar app.jar

rem Pre-launch checks on Windows
rem 1. System commit headroom: (Get-CimInstance Win32_OperatingSystem).FreePhysicalMemory and FreeVirtualMemory should be well above a few hundred MB
rem 2. Prefer 64-bit java.exe: where java   -> confirm path is under a 64-bit JDK
rem 3. Job Object / container caps: verify the memory quota exceeds -Xmx + ~512MB
java -agentlib:... -javaagent:C:\agents\agent.jar -jar app.jar

// Native malloc failure inside libinstrument is not a Java exception; catch at the process level:
// launch the JVM from a wrapper that checks the exit code and memory counters, then reports environment failure.

When it happens

Trigger: Running java -javaagent:agent.jar on Windows when the native heap cannot satisfy a tiny allocation: process working set/commit limit exhausted, Job Object memory caps, or severe native fragmentation from other DLLs loaded at startup.

Common situations: Windows containers or Job Objects with tight commit limits; machines under heavy memory pressure; AV/EDR software injecting DLLs that consume native memory before the JVM starts; 32-bit JVMs near the 2/4GB address-space ceiling.

Related errors

• OOM error in native tmp buffer allocation
• -javaagent: memory allocation failure.\n
• -javaagent: memory allocation failed\n
• java.lang.instrument/-javaagent: cannot create native agent.
• java.lang.instrument/-javaagent: initialization of native ag

AI-assisted analysis of openjdk/jdk@88dfb74bbe (2026-08-14). Data as JSON: /api/errors/ee09cc5577300972. Report an issue: GitHub.