What causes "All parts but the last must be larger than %d bytes" in apache/beam?

Increase the chunk size so every part except the last is >= MIN_PART_SIZE (5 MiB)

apache/beam · error · S3ClientError

All parts but the last must be larger than

Error message

All parts but the last must be larger than %d bytes

What it means

FakeS3Client.complete_multipart_upload raises S3ClientError('All parts but the last must be larger than MIN_PART_SIZE bytes', 400) when any non-final part is smaller than the minimum part size (5 MiB in real S3). It mirrors S3's EntityTooSmall error.

Solutions

Increase the chunk size so every part except the last is >= MIN_PART_SIZE (5 MiB)
Merge undersized parts together before completing the upload
Use fewer parts for small datasets, or skip multipart for small payloads
Adjust MIN_PART_SIZE in the fake only for tiny test fixtures, acknowledging the divergence from real S3

Example fix

// before
parts = [data[i:i+1024*1024] for i in range(0, len(data), 1024*1024)]  # 1 MiB < 5 MiB min
// after
MIN_PART_SIZE = 5 * 1024 * 1024
parts = [data[i:i+MIN_PART_SIZE] for i in range(0, len(data), MIN_PART_SIZE)]

Defensive patterns

Strategy: validation

Validate before calling

sizes = [len(p) for p in parts]
assert all(s >= fake_client.MIN_PART_SIZE for s in sizes[:-1]), f'undersized non-final parts: {sizes}'

Prevention

Use a chunk size >= 5 MiB for all but the last part
Merge small parts before completing the upload
Skip multipart entirely for small payloads
Account for MIN_PART_SIZE when generating test data

When it happens

Trigger: Completing a multipart upload where any part except the last has len(bytes) < MIN_PART_SIZE, typically because data was chunked too finely.

Common situations: Small test datasets split into too many tiny parts; a chunk-size configuration set below 5 MiB; merging logic that splits on record boundaries producing small shards.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5a1b40a115029231. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/clients/s3/fake_client.py:226

    # Check that we got all the parts that they intended to send
    part_numbers_to_confirm = set(part['PartNumber'] for part in request.parts)

    # Make sure all the expected parts are present
    if part_numbers_to_confirm != set(parts_received.keys()):
      raise messages.S3ClientError(
          'One or more of the specified parts could not be found', 400)

    # Sort by part number
    sorted_parts = sorted(parts_received.items(), key=lambda pair: pair[0])
    sorted_bytes = [bytes_ for (_, bytes_) in sorted_parts]

    # Make sure that the parts aren't too small (except the last part)
    part_sizes = [len(bytes_) for bytes_ in sorted_bytes]
    if any(size < MIN_PART_SIZE for size in part_sizes[:-1]):
      e_message = """
      All parts but the last must be larger than %d bytes
      """ % MIN_PART_SIZE
      raise messages.S3ClientError(e_message, 400)

    # String together all bytes for the given upload
    final_contents = b''.join(sorted_bytes)

    # Create FakeFile object
    num_parts = len(parts_received)
    etag = '"%s-%d"' % ('x' * 32, num_parts)
    file_ = FakeFile(request.bucket, request.object, final_contents, etag=etag)

    # Store FakeFile in self.files
    self.add_file(file_)

View on GitHub (pinned to 12126d8942)