{"record":{"id":"c4bc56caefd2daaa","repo":"apache/hadoop","slug":"total-heap-the-can-be-used-is-bytes-while-the-e","errorCode":null,"errorMessage":"Total heap the can be used is {} bytes while the emulator is configured to emulate a total of {} bytes","messagePattern":"Total heap the can be used is (.+?) bytes while the emulator is configured to emulate a total of (.+?) bytes","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/TotalHeapUsageEmulatorPlugin.java","lineNumber":166,"sourceCode":"    /**\n     * Gets the total number of 1mb objects stored in the emulator.\n     *\n     * @return total number of 1mb objects.\n     */\n    @VisibleForTesting\n    public int getHeapSpaceSize() {\n      return heapSpace.size();\n    }\n\n    /**\n     * This will initialize the core and check if the core can emulate the \n     * desired target on the underlying hardware.\n     */\n    public void initialize(ResourceCalculatorPlugin monitor, \n                           long totalHeapUsageInMB) {\n      long maxPhysicalMemoryInMB = monitor.getPhysicalMemorySize() / ONE_MB ;\n      if(maxPhysicalMemoryInMB < totalHeapUsageInMB) {\n        throw new RuntimeException(\"Total heap the can be used is \" \n            + maxPhysicalMemoryInMB \n            + \" bytes while the emulator is configured to emulate a total of \" \n            + totalHeapUsageInMB + \" bytes\");\n      }\n    }\n    \n    /**\n     * Clear references to all the GridMix-allocated special objects so that \n     * heap usage is reduced.\n     */\n    @Override\n    public void reset() {\n      heapSpace.clear();\n    }\n  }\n  \n  public TotalHeapUsageEmulatorPlugin() {\n    this(new DefaultHeapUsageEmulator());","sourceCodeStart":148,"sourceCodeEnd":184,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/TotalHeapUsageEmulatorPlugin.java#L148-L184","documentation":"TotalHeapUsageEmulatorPlugin.initialize(monitor, totalHeapUsageInMB) compares the machine's physical memory (monitor.getPhysicalMemorySize() / ONE_MB) against the requested emulated heap usage in MB and throws RuntimeException if the target exceeds physical memory — the emulator allocates real objects, so it cannot emulate more heap than the box physically has. The message misleadingly says 'bytes' although both numbers are megabytes.","triggerScenarios":"Configuring gridmix heap emulation with an emulated total-heap target (from the trace or gridmix.emulators.resource-usage.heap.* settings) larger than the physical RAM of the node running the task, e.g. a trace captured on a 512GB node replayed on a 16GB node with heap emulation enabled.","commonSituations":"Replaying production traces on smaller dev/test hardware; container memory limits conflated with physical memory (the check uses getPhysicalMemorySize, not container size); the 'bytes' vs 'MB' wording in the message confusing operators into misjudging the shortfall.","solutions":["Lower the emulated heap target (scale the trace's heap numbers down to fit the replay node, e.g. via a scale factor) so totalHeapUsageInMB <= physical memory MB","Run gridmix on nodes with at least as much physical RAM as the largest emulated heap","Disable heap emulation by removing TotalHeapUsageEmulatorPlugin from gridmix.emulators.resource-usage.plugins if heap fidelity is not required","Read the two numbers in the message as MB despite the word 'bytes'"],"exampleFix":"# before (on a 16GB node)\n-Dgridmix.emulators.resource-usage.plugins=org.apache.hadoop.mapred.gridmix.emulators.resourceusage.TotalHeapUsageEmulatorPlugin\n# trace job wants 64GB heap -> RuntimeException\n\n# after: scale gridmix down to the node\n-Dgridmix.submit.multiplier.interval=... \n# or scale heap targets (e.g. gridmix scale 0.1) / drop the plugin:\n-Dgridmix.emulators.resource-usage.plugins=org.apache.hadoop.mapred.gridmix.emulators.resourceusage.CumulativeCpuUsageEmulatorPlugin","handlingStrategy":"validation","validationCode":"// Before running gridmix with heap emulation\nlong physicalMB = new LinuxResourceCalculatorPlugin().getPhysicalMemorySize() / (1024 * 1024);\nif (targetHeapMB > physicalMB) {\n  targetHeapMB = (long) (physicalMB * 0.8); // scale down to fit node\n}","typeGuard":null,"tryCatchPattern":"try {\n  heapPlugin.initialize(monitor, totalHeapUsageInMB);\n} catch (RuntimeException e) {\n  // message numbers are MB despite saying 'bytes'; lower the emulated heap target\n}","preventionTips":["Scale trace-derived heap targets to the replay cluster before enabling heap emulation","Compare emulated heap against the node's physical RAM, not container limits — that is what the plugin checks"],"tags":["hadoop","gridmix","heap-emulation","resource-emulation","capacity"],"backgroundTag":"insufficient-memory","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}