apache/beam · error · IOException

Could not create a temporary directory for storing dependenc

Error message

Could not create a temporary directory for storing dependencies: {dependenciesDir.getAbsolutePath()}

What it means

Thrown by TransformServiceLauncher when it fails to create the temporary local directory used to stage extra dependencies (pythonRequirementsFile staging dir) before launching the Beam Transform Service. File.mkdir() returns false (instead of throwing) when the parent directory does not exist or the process lacks write permission on it, and the launcher converts that into this IOException. Without this directory the launcher cannot write the updated requirements file or the DEPENDENCIES_VOLUME mount, so startup is aborted.

Source

Thrown at sdks/java/transform-service/launcher/src/main/java/org/apache/beam/sdk/transformservice/launcher/TransformServiceLauncher.java:140

        LOG.error(
            "GCP credentials will not be available for the transform service since the Google "
                + "Cloud application default credentials file could not be found at the expected "
                + "location {}.",
            applicationDefaultFilePath);
      }
    }

    // Setting up the dependencies directory.
    File dependenciesDir = Paths.get(tmpDir, "dependencies_dir").toFile();
    Path updatedRequirementsFilePath = Paths.get(dependenciesDir.toString(), "requirements.txt");
    if (dependenciesDir.exists()) {
      LOG.info("Reusing the existing dependencies directory {}", dependenciesDir.getAbsolutePath());
    } else {
      LOG.info(
          "Creating a temporary directory for storing dependencies: {}",
          dependenciesDir.getAbsolutePath());
      if (!dependenciesDir.mkdir()) {
        throw new IOException(
            "Could not create a temporary directory for storing dependencies: "
                + dependenciesDir.getAbsolutePath());
      }

      // We create a requirements file with extra dependencies.
      // If there are no extra dependencies, we just provide an empty requirements file.
      File file = updatedRequirementsFilePath.toFile();
      if (!file.createNewFile()) {
        throw new IOException(
            "Could not create the new requirements file " + updatedRequirementsFilePath);
      }

      // Updating dependencies.
      if (pythonRequirementsFile != null) {
        Path requirementsFilePath = Paths.get(pythonRequirementsFile);
        List<String> updatedLines = new ArrayList<>();

        try (Stream<String> lines = java.nio.file.Files.lines(requirementsFilePath)) {

View on GitHub (pinned to 12126d8942)

Solutions

Create the parent directory of dependenciesDir first (Files.createDirectories on the parent), since mkdir() does not create intermediate directories.
Check filesystem permissions/write access for the user running the launcher and pick a writable location (e.g. java.io.tmpdir).
Verify the volume backing the path is writable (not read-only mount, not out of disk space).

Example fix

// before: parent does not exist, mkdir() fails
Path deps = Paths.get("/opt/beam/transforms/deps");
service.launch(deps);

// after: ensure parents exist first
Files.createDirectories(Paths.get("/opt/beam/transforms/deps"));

Defensive patterns

Strategy: validation

Validate before calling

Path deps = Paths.get(dependenciesDirPath);
if (java.nio.file.Files.exists(deps) && !java.nio.file.Files.isDirectory(deps))
  throw new IllegalStateException("dependencies path is not a directory");
if (!java.nio.file.Files.exists(deps))
  java.nio.file.Files.createDirectories(deps.getParent() == null ? deps : deps.getParent());
if (!java.nio.file.Files.isWritable(java.nio.file.Files.exists(deps) ? deps : deps.getParent()))
  throw new IllegalStateException("no write permission for dependencies directory");

Prevention

Use Files.createDirectories (creates parents) instead of relying on mkdir().
Run the launcher as a user with write access to the staging location.
Prefer java.io.tmpdir or another guaranteed-writable volume in containers.

When it happens

Trigger: Calling TransformServiceLauncher's public startup path (e.g. via main/expand flows) when dependenciesDir already does not exist AND File.mkdir() fails: the parent directory of dependenciesDir is missing, the filesystem is read-only, or the user lacks write permission at that location.

Common situations: Running the launcher in a container or CI environment where the default temp path is read-only or mounted noexec; a custom dependencies directory passed whose parent does not exist; running as a non-root user without write access to the configured directory; disk full on the staging volume.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a841f677521a3046. Report an issue: GitHub.