apache/beam · error · IOError

err (re-raised OSError from os.makedirs)

Error message

err (re-raised OSError from os.makedirs)

What it means

LocalFileSystem.mkdirs wraps os.makedirs and converts any OSError (e.g. the leaf directory already exists, or a permission problem) into a plain IOError with the original error as its argument. It is the local implementation of the Beam FileSystem directory-creation contract, which documents IOError as the failure mode.

Source

Thrown at sdks/python/apache_beam/io/localfilesystem.py:81

      path: path as a string
    Returns:
      a pair of path components as strings.
    """
    return os.path.split(os.path.abspath(path))

  def mkdirs(self, path):
    """Recursively create directories for the provided path.

    Args:
      path: string path of the directory structure that should be created

    Raises:
      IOError: if leaf directory already exists.
    """
    try:
      os.makedirs(path)
    except OSError as err:
      raise IOError(err)

  def has_dirs(self):
    """Whether this FileSystem supports directories."""
    return True

  def _url_dirname(self, url_or_path):
    """Pass through to os.path.dirname.

    This version uses os.path instead of posixpath to be compatible with the
    host OS.

    Args:
      url_or_path: A string in the form of /some/path.
    """
    return os.path.dirname(url_or_path)

  def _list(self, dir_or_prefix):
    """List files in a location.

View on GitHub (pinned to 12126d8942)

Solutions

1. Check os.path.exists(path) and skip mkdirs if the directory is already present, or wrap the call in try/except IOError and ignore FileExistsError cases.
2. Verify the process has write permission on the parent directory, or run with appropriate credentials/ownership.
3. Make sure the path passed is a directory path, not an existing file path.
4. Prefer beam.io.filesystems.FileSystems.mkdirs for URL-based paths so the correct filesystem implementation handles the path.

Example fix

// before
fs.mkdirs(path)  # crashes if leaf dir exists
// after
import os
if not os.path.exists(path):
    fs.mkdirs(path)

import os
if not os.path.exists(path):
    os.makedirs(path, exist_ok=True)  # or fs.mkdirs(path)

try:
    fs.mkdirs(path)
except IOError as e:
    if not os.path.isdir(path):
        raise  # ignore benign already-exists; surface real errors

When it happens

Trigger: Calling filesystem.mkdirs(path) where the leaf directory already exists (FileExistsError), a parent component is a file (NotADirectoryError), or the process lacks write permission on the parent (PermissionError). Any of these OS-level failures surface as IOError(err).

Common situations: Racing workers both creating the same temp/staging directory; writing to a read-only mount or a path owned by another user; accidentally passing a file path instead of a directory path.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

• List operation failed
• err (re-raised OSError during copy)
• Copy operation failed
• err (re-raised OSError from os.rename)
• Rename operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3c942f19d3df5a88. Report an issue: GitHub.