microsoft/semantic-kernel · error · ValueError

Batch upsert failed

Error message

Batch upsert failed

What it means

Raised as a ValueError in MongoDBAtlasMemoryStore.upsert_batch when the sum of matched_count and upserted_count from the bulk_write result does not equal the number of records submitted. The store uses bulk_write with ordered=False and UpdateOne(upsert=True); each record should match or upsert exactly once, so a mismatch indicates some records were neither matched nor inserted.

Source

Thrown at python/semantic_kernel/connectors/memory_stores/mongodb_atlas/mongodb_atlas_memory_store.py:193

        Returns:
            List[str]: The unique identifiers for the memory records.
        """
        upserts: list[UpdateOne] = []
        for record in records:
            document = memory_record_to_mongo_document(record)
            upserts.append(UpdateOne(document, {"$set": document}, upsert=True))
        bulk_update_result: results.BulkWriteResult = await self.database[collection_name].bulk_write(
            upserts, ordered=False
        )

        # Assert the number matched and the number upserted equal the total batch updated
        logger.debug(
            "matched_count=%s, upserted_count=%s",
            bulk_update_result.matched_count,
            bulk_update_result.upserted_count,
        )
        if bulk_update_result.matched_count + bulk_update_result.upserted_count != len(records):
            raise ValueError("Batch upsert failed")
        return [record._id for record in records]

    async def get(self, collection_name: str, key: str, with_embedding: bool) -> MemoryRecord:
        """Gets a memory record from the data store. Does not guarantee that the collection exists.

        Args:
            collection_name (str): The name associated with a collection of embeddings.
            key (str): The unique id associated with the memory record to get.
            with_embedding (bool): If true, the embedding will be returned in the memory record.

        Returns:
            MemoryRecord: The memory record if found
        """
        document = await self.database[collection_name].find_one({MONGODB_FIELD_ID: key})

        return document_to_memory_record(document, with_embedding) if document else None

    async def get_batch(self, collection_name: str, keys: list[str], with_embeddings: bool) -> list[MemoryRecord]:

View on GitHub (pinned to c028a0c7dc)

Solutions

1. Inspect the BulkWriteResult (matched_count, upserted_count, write_errors) by performing the bulk_write manually to see which documents failed.
2. Check for collection-level schema validation rules that reject some records.
3. Ensure each record produces a unique, valid document with a well-formed _id.
4. Reduce the batch and retry to isolate the offending record(s).

Example fix

// before
ids = await store.upsert_batch('docs', records)  # ValueError: Batch upsert failed
// after
try:
    ids = await store.upsert_batch('docs', records)
except ValueError:
    # fall back to single upserts to isolate failures
    ids = []
    for r in records:
        try:
            ids.append(await store.upsert('docs', r))
        except Exception as e:
            logging.error('Record upsert failed: %s', e)

def documents_have_unique_ids(records) -> bool:
    ids = [r._id for r in records]
    return len(ids) == len(set(ids))

if not documents_have_unique_ids(records):
    raise ValueError('Duplicate ids in batch')

try:
    ids = await store.upsert_batch('docs', records)
except ValueError:
    # isolate failing records by falling back to single upserts
    ids = []
    for r in records:
        try:
            ids.append(await store.upsert('docs', r))
        except Exception as e:
            logging.error('Record failed: %s', e)

When it happens

Trigger: A bulk write where some operations fail validation server-side (e.g. document schema validation rules in Atlas), are dropped, or produce duplicate-key conflicts in an unordered bulk. The counts diverge from the input batch size.

Common situations: MongoDB collection-level JSON schema validation rejecting some documents. Duplicate _id conflicts causing some UpdateOne ops to not count as matched or upserted. Network issues causing partial bulk completion. Documents missing required fields per a validator.

Related errors

• Upsert failed
• Collection '{collection_name}' does not exist
• Collection {collection_name} does not exist, cannot insert.
• Upsert failed due to: {e}
• Failed to create MongoDB Atlas settings.

AI-assisted analysis of microsoft/semantic-kernel@c028a0c7dc (2026-08-13). Data as JSON: /api/errors/d840d4ea3a812855. Report an issue: GitHub.