apache/beam · error · ValueError

Invalid page token.

Error message

Invalid page token.

What it means

The fake (in-memory) S3 client used by apache_beam tests raises ValueError('Invalid page token.') in list() when a ListRequest carries a continuation_token that is not present in its internal token registry. This simulates S3 rejecting an expired or fabricated pagination token; the fake deletes tokens after one use, so replaying a token also fails.

Source

Thrown at sdks/python/apache_beam/io/aws/clients/s3/fake_client.py:114

    matching_files = []

    for file_bucket, file_name in sorted(iter(self.files)):
      if bucket == file_bucket and file_name.startswith(prefix):
        file_object = self.get_file(file_bucket, file_name).get_metadata()
        matching_files.append(file_object)

    if not matching_files:
      message = 'Tried to list nonexistent S3 path: s3://%s/%s' % (
          bucket, prefix)
      raise messages.S3ClientError(message, 404)

    # Handle pagination.
    items_per_page = 5
    if not request.continuation_token:
      range_start = 0
    else:
      if request.continuation_token not in self.list_continuation_tokens:
        raise ValueError('Invalid page token.')
      range_start = self.list_continuation_tokens[request.continuation_token]
      del self.list_continuation_tokens[request.continuation_token]

    result = messages.ListResponse(
        items=matching_files[range_start:range_start + items_per_page])

    if range_start + items_per_page < len(matching_files):
      next_range_start = range_start + items_per_page
      next_continuation_token = '_page_token_%s_%s_%d' % (
          bucket, prefix, next_range_start)
      self.list_continuation_tokens[next_continuation_token] = next_range_start
      result.next_token = next_continuation_token

    return result

  def get_range(self, request, start, end):
    r"""Retrieves an object.

View on GitHub (pinned to 12126d8942)

Solutions

1. Use only continuation tokens taken from the immediately preceding ListResponse of the same fake client instance.
2. Do not reuse a token after a successful page fetch — the fake deletes consumed tokens.
3. Recreate/restart pagination from the beginning (no continuation_token) if the fake client was reset.
4. In tests, drive pagination in a loop using response.next_token rather than fixed tokens.

Example fix

// before
client.list(messages.ListRequest(continuation_token='tok-from-old-instance'))
// after
resp = client.list(messages.ListRequest())
next_req = messages.ListRequest(continuation_token=resp.next_token)  # token from same client

def token_is_fresh(token: str | None, client) -> bool:
    return token is None or token in client.list_continuation_tokens

def has_valid_token(req) -> bool:
    return not getattr(req, 'continuation_token', None) or \
           req.continuation_token in client.list_continuation_tokens

try:
    resp = client.list(request)
except ValueError as e:
    if str(e) == 'Invalid page token.':
        request.continuation_token = None  # restart pagination
        resp = client.list(request)
    else:
        raise

When it happens

Trigger: Calling FakeAwsS3Client.list() with request.continuation_token that was never issued, was already consumed (fake deletes tokens after use), or the fake client was recreated between pages so the registry is empty.

Common situations: Unit tests replaying a saved continuation token; retrying a page after the fake was reset; holding tokens across two different fake client instances; tests hard-coding token strings.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• error retrieving page: %v
• Basepath %r must be S3 path.
• Path %r must be S3 path.
• Invalid path: %s
• List operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e977bfdcaa77a5d0. Report an issue: GitHub.