apache/beam · error · BeamIOError

last_updated operation failed

Error message

last_updated operation failed

What it means

Raised by S3FileSystem.last_updated when S3IO.last_updated(path) raises; wrapped as BeamIOError keyed by path. Fetching the modification timestamp requires the object to exist and a working S3 client.

Solutions

1. Confirm the object exists (fs.exists) before querying last_updated
2. Verify IAM permissions include s3:GetObject/s3:ListBucket
3. Check AWS credentials and client region configuration
4. Catch BeamIOError and skip/handle paths whose metadata is unavailable

Example fix

// before
mtime = fs.last_updated(path)
// after
try:
    mtime = fs.last_updated(path)
except BeamIOError as e:
    logging.warning('no mtime for %s: %s', path, e)
    mtime = None

if not fs.exists(path):
    raise FileNotFoundError(path)

try:
    mtime = fs.last_updated(path)
except BeamIOError as e:
    log.warning('last_updated failed for %s: %s', path, e)
    mtime = None

When it happens

Trigger: Calling fs.last_updated('s3://bucket/key') on a nonexistent object, malformed path, or when the S3 HEAD request fails due to credentials, region, or network errors.

Common situations: Incremental sync jobs reading mtimes of files that were deleted or renamed; bucket in wrong region relative to client config; missing IAM s3:GetObject permission.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Metadata operation failed
• Basepath %r must be S3 path.
• Checksum operation failed
• Delete operation failed
• ENOENT

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e4df795cf26d55e0. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/s3filesystem.py:266

      return s3io.S3IO(options=self._options).size(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("size() operation failed", {path: e})

  def last_updated(self, path):
    """Get UNIX Epoch time in seconds on the FileSystem.

    Args:
      path: string path of file.

    Returns: float UNIX Epoch time

    Raises:
      ``BeamIOError``: if path doesn't exist.
    """
    try:
      return s3io.S3IO(options=self._options).last_updated(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("last_updated operation failed", {path: e})

  def checksum(self, path):
    """Fetch checksum metadata of a file on the
    :class:`~apache_beam.io.filesystem.FileSystem`.

    Args:
      path: string path of a file.

    Returns: string containing checksum

    Raises:
      ``BeamIOError``: if path isn't a file or doesn't exist.
    """
    try:
      return s3io.S3IO(options=self._options).checksum(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("Checksum operation failed", {path: e})

View on GitHub (pinned to 12126d8942)