apache/beam · error · BeamIOError

List operation failed

Error message

List operation failed

What it means

BlobStorageFileSystem._list wraps the underlying BlobStorageIO.list_files iteration; any exception (auth failure, missing container, network error, bad prefix) is caught and re-raised as BeamIOError('List operation failed') with the failing path and original exception attached to exception_details.

Solutions

1. Inspect BeamIOError.exception_details[path] for the root cause exception.
2. Verify the container exists and the prefix is correct.
3. Check Azure credentials (connection string / account key / Entra token) are present and valid in the runner environment.
4. Retry on transient errors; configure storage firewall/network rules to allow the job's egress.

Example fix

// before
for m in fsys.match(['az://wrong-container/*']): ...
// after
try:
    for m in fsys.match(['az://container/*']): ...
except BeamIOError as e:
    for path, err in e.exception_details.items():
        logging.error('list failed for %s: %s', path, err)

from azure.storage.blob import BlobServiceClient
try:
    BlobServiceClient(account_url=f"https://{account}.blob.core.windows.net", credential=cred).get_container_client(container).get_account_information()
except Exception as e:
    raise RuntimeError(f"Azure list precheck failed: {e}")

def list_root_cause(exc):
    return getattr(exc, 'exception_details', {}).get(path) if (exc := exc) else None

try:
    metas = fsys.metadata(['az://container/prefix'])
except BeamIOError as e:
    for path, cause in e.exception_details.items():
        logging.error("list failed for %s: %r", path, cause)
    if is_transient(cause):
        retry(...)

When it happens

Trigger: Calling match_files/list_prefix on a container that does not exist, with missing/misconfigured Azure credentials (AZURE_STORAGE_ACCOUNT / AZURE_STORAGE_KEY or token), or during a transient network failure.

Common situations: Wrong container name in the path; DefaultAzureCredential chain exhausted in CI; storage account firewall blocking the runner; typo in prefix producing 404 from the service.

Related errors

• Basepath %r must be an Azure Blob Storage path.
• Invalid path
• Path %r must be Azure Blob Storage path.
• Unable to copy unequal number of sources and destinations.
• delete does not delete containers.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/101adb96254238c7. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/azure/blobstoragefilesystem.py:130

    return False

  def _list(self, dir_or_prefix):
    """List files in a location.
    Listing is non-recursive (for filesystems that support directories).
    Args:
      dir_or_prefix: (string) A directory or location prefix (for filesystems
        that don't have directories).
    Returns:
      Generator of ``FileMetadata`` objects.
    Raises:
      ``BeamIOError``: if listing fails, but not if no files were found.
    """
    try:
      for path, (size, updated) in self._blobstorageIO().list_files(
          dir_or_prefix, with_metadata=True):
        yield FileMetadata(path, size, updated)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("List operation failed", {dir_or_prefix: e})

  def _blobstorageIO(self):
    return blobstorageio.BlobStorageIO(pipeline_options=self._pipeline_options)

  def _path_open(
      self,
      path,
      mode,
      mime_type='application/octet-stream',
      compression_type=CompressionTypes.AUTO):
    """Helper functions to open a file in the provided mode.
    """
    compression_type = FileSystem._get_compression_type(path, compression_type)
    mime_type = CompressionTypes.mime_type(compression_type, mime_type)
    raw_file = self._blobstorageIO().open(path, mode, mime_type=mime_type)
    if compression_type == CompressionTypes.UNCOMPRESSED:
      return raw_file
    return CompressedFile(raw_file, compression_type=compression_type)

View on GitHub (pinned to 12126d8942)