What causes "Unable to delete local heap dump {}" in apache/beam?

Ensure the worker process user has write/delete permission on the heap dump directory.

Unable to delete local heap dump

Error message

Unable to delete local heap dump {}

What it means

Warning logged when Files.delete on the local heap dump file fails with IOException after an upload attempt. The dump was (or was not) uploaded but could not be cleaned up from local disk, potentially wasting worker disk space. Upload outcome is unaffected; the method returns based on the upload result, not the delete.

Solutions

Ensure the worker process user has write/delete permission on the heap dump directory.
Free up disk space if repeated undeleted dumps are filling the disk; reduce heap dump frequency.
Manually delete stale heap_dump*.hprof files on workers if this recurs; upgrade Beam to pick up fixes in dump cleanup.

Example fix

// ensure dump folder is writable by the harness user
chmod 777 /tmp/heapdumps   // or run the worker with an appropriate UID

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-check writability of dump location
if (!Files.isWritable(Paths.get(dumpFolder))) { /* relocate dump folder */ }

Try / catch

try { Files.delete(localDump); } catch (IOException e) { LOG.warn("manual cleanup of {} needed: {}", localDump, e.getMessage()); }

Prevention

Grant the worker user delete permissions on the dump directory
Avoid external agents that hold .hprof files open
Periodically clean stale heap dumps from worker volumes

When it happens

Trigger: After the upload block, Files.delete(localSource.toPath()) throws IOException: file locked by another process, permission denied, or the file was concurrently removed by the JVM or an external cleaner.

Common situations: Workers with restricted tmpdir permissions, antivirus/agent processes holding the .hprof open, or small ephemeral disks filling up from repeated undeleted dumps.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a65a5e754892da8a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/harness/src/main/java/org/apache/beam/fn/harness/status/MemoryMonitor.java:361

        LOG.warn(
            "Heap dump {} of {}MB detected, attempting to upload file to {}",
            localSource,
            localSource.length() / 1024 / 1024,
            remoteDest);
        ResourceId resource = FileSystems.matchNewResource(remoteDest, false);
        try {
          uploadFile(localSource, resource, gzipCompress);
          uploadedHeapDump = true;
          LOG.warn("Heap dump {} uploaded to {}", localSource, remoteDest);
        } catch (IOException e) {
          LOG.error("Error uploading heap dump to {}", remoteDest, e);
        }

        try {
          Files.delete(localSource.toPath());
          LOG.info("Deleted local heap dump {}", localSource);
        } catch (IOException e) {
          LOG.warn("Unable to delete local heap dump {}", localSource, e);
        }
      }
    }

    return uploadedHeapDump;
  }

  private void uploadFile(File srcPath, ResourceId destination, boolean gzipCompress)
      throws IOException {
    StandardCreateOptions createOptions =
        StandardCreateOptions.builder().setMimeType("application/octet-stream").build();
    try (WritableByteChannel dst = FileSystems.create(destination, createOptions);
        ReadableByteChannel src = Channels.newChannel(new FileInputStream(srcPath))) {
      if (gzipCompress) {
        try (OutputStream os = new GZIPOutputStream(Channels.newOutputStream(dst));
            WritableByteChannel gzipDst = Channels.newChannel(os)) {
          ByteStreams.copy(src, gzipDst);
        }

View on GitHub (pinned to 12126d8942)