apache/beam · error · S3ClientError

One or more of the specified parts could not be found

Error message

One or more of the specified parts could not be found

What it means

FakeS3Client.complete_multipart_upload raises S3ClientError('One or more of the specified parts could not be found', 400) when the part numbers listed in the complete request don't exactly match the parts actually received via upload_part. It mirrors the InvalidPart error from real S3.

Solutions

1. Make the completed parts list exactly match the (part_number, etag) pairs returned by each upload_part call
2. Track uploaded part numbers in a set and pass that to the complete request
3. Remove duplicates and fill in any missing part numbers before completing
4. Log/compare requested vs received part numbers when this error occurs in tests

Example fix

// before
fake_client.complete_multipart_upload(messages.CompleteMultipartUploadRequest(upload_id, [{'PartNumber': 1}, {'PartNumber': 3}]))
// after
parts = [{'PartNumber': n, 'ETag': etag} for n, etag in uploaded_parts.items()]
fake_client.complete_multipart_upload(messages.CompleteMultipartUploadRequest(upload_id, parts))

received = set(fake_client.multipart_uploads[upload_id].keys())
requested = {p['PartNumber'] for p in parts}
assert requested == received, f'mismatch: missing={received-requested}, extra={requested-received}'

When it happens

Trigger: Completing an upload where request.parts references part numbers never uploaded (or vice versa): skipped a part, uploaded an extra part, or duplicate part numbers in the request.

Common situations: Retries that re-uploaded some parts and the completed list diverged; loop skipping an index; building the parts list from metadata out of sync with what was actually sent.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• All parts but the last must be larger than
• Param validation failed on part number
• The specified upload does not exist
• AWS dependencies are not installed, and no alternative…
• Basepath %r must be S3 path.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c1e7377d557c0530. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/clients/s3/fake_client.py:213

    if upload_id not in self.multipart_uploads:
      raise messages.S3ClientError('The specified upload does not exist', 404)

    self.multipart_uploads[upload_id][part_number] = request.bytes

    etag = '"%s"' % ('x' * 32)
    return messages.UploadPartResponse(etag, part_number)

  def complete_multipart_upload(self, request):
    MIN_PART_SIZE = 5 * 2**10  # 5 KiB

    parts_received = self.multipart_uploads[request.upload_id]

    # Check that we got all the parts that they intended to send
    part_numbers_to_confirm = set(part['PartNumber'] for part in request.parts)

    # Make sure all the expected parts are present
    if part_numbers_to_confirm != set(parts_received.keys()):
      raise messages.S3ClientError(
          'One or more of the specified parts could not be found', 400)

    # Sort by part number
    sorted_parts = sorted(parts_received.items(), key=lambda pair: pair[0])
    sorted_bytes = [bytes_ for (_, bytes_) in sorted_parts]

    # Make sure that the parts aren't too small (except the last part)
    part_sizes = [len(bytes_) for bytes_ in sorted_bytes]
    if any(size < MIN_PART_SIZE for size in part_sizes[:-1]):
      e_message = """
      All parts but the last must be larger than %d bytes
      """ % MIN_PART_SIZE
      raise messages.S3ClientError(e_message, 400)

    # String together all bytes for the given upload
    final_contents = b''.join(sorted_bytes)

    # Create FakeFile object

View on GitHub (pinned to 12126d8942)