apache/beam · error · IOException

Could not create the new requirements file

Error message

Could not create the new requirements file {updatedRequirementsFilePath}

What it means

Thrown when the launcher cannot create the updated Python requirements file inside the dependencies directory. File.createNewFile() returns false if the file already exists (or fails atomically), and the launcher raises this IOException. This file is required to feed extra dependencies to the transform service container via PYTHON_REQUIREMENTS_FILE_NAME, so the launch is aborted.

Solutions

1. Delete stale files in the dependencies directory (or use a fresh temporary directory) before relaunching.
2. Ensure only one launcher instance uses the same dependencies directory at a time.
3. If the file legitimately exists and is current, treat it as success rather than an error path — i.e. clean the directory before startup.

Example fix

// before: reused dir already contains requirements file
// launcher: if (!file.createNewFile()) throw ...

// after: caller cleans staging dir before launch
FileUtils.deleteDirectory(dependenciesDir);
dependenciesDir.mkdir();

File req = updatedRequirementsFilePath.toFile();
if (req.exists()) {
  // stale artifact from a previous run
  java.nio.file.Files.delete(req.toPath());
}

When it happens

Trigger: Calling the launcher's startup path when updatedRequirementsFilePath.toFile().createNewFile() returns false — typically because a file already exists at that path from a previous run in a reused dependencies directory.

Common situations: Re-running the launcher with a persistent/reused dependencies directory where a stale requirements file from an earlier launch still exists; two launcher processes racing on the same staging directory.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

• Could not create a temporary directory for storing…
• Could not create a temporary directory for storing…
• Error matching file spec
• Error when finding
• Failed to create directory:

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/04e5a0e57028348a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/transform-service/launcher/src/main/java/org/apache/beam/sdk/transformservice/launcher/TransformServiceLauncher.java:149

    File dependenciesDir = Paths.get(tmpDir, "dependencies_dir").toFile();
    Path updatedRequirementsFilePath = Paths.get(dependenciesDir.toString(), "requirements.txt");
    if (dependenciesDir.exists()) {
      LOG.info("Reusing the existing dependencies directory {}", dependenciesDir.getAbsolutePath());
    } else {
      LOG.info(
          "Creating a temporary directory for storing dependencies: {}",
          dependenciesDir.getAbsolutePath());
      if (!dependenciesDir.mkdir()) {
        throw new IOException(
            "Could not create a temporary directory for storing dependencies: "
                + dependenciesDir.getAbsolutePath());
      }

      // We create a requirements file with extra dependencies.
      // If there are no extra dependencies, we just provide an empty requirements file.
      File file = updatedRequirementsFilePath.toFile();
      if (!file.createNewFile()) {
        throw new IOException(
            "Could not create the new requirements file " + updatedRequirementsFilePath);
      }

      // Updating dependencies.
      if (pythonRequirementsFile != null) {
        Path requirementsFilePath = Paths.get(pythonRequirementsFile);
        List<String> updatedLines = new ArrayList<>();

        try (Stream<String> lines = java.nio.file.Files.lines(requirementsFilePath)) {
          lines.forEachOrdered(
              line -> {
                Path dependencyFilePath = Paths.get(line);
                if (java.nio.file.Files.exists(dependencyFilePath)) {
                  Path fileName = dependencyFilePath.getFileName();
                  if (fileName == null) {
                    throw new IllegalArgumentException(
                        "Could not determine the filename of the local artifact "
                            + dependencyFilePath);

View on GitHub (pinned to 12126d8942)