{"record":{"id":"d9c2d5788c28c23c","repo":"RyanCodrai/turbovec","slug":"failed-to-embed-len-missing-document-s-ids","errorCode":null,"errorMessage":"failed to embed {len(missing)} document(s): {ids}","messagePattern":"failed to embed (.+?) document\\(s\\): (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"turbovec-python/python/turbovec/agno.py","lineNumber":501,"sourceCode":"            for doc in documents:\n                meta = dict(doc.meta_data) if doc.meta_data else {}\n                meta.update(filters)\n                doc.meta_data = meta\n\n        self._embed_missing(documents)\n\n        # Raise on any document that still lacks an embedding rather than\n        # silently dropping — silent drops mask data-pipeline bugs.\n        # None/len check instead of truthiness: `not <ndarray>` raises the\n        # numpy truth-value-ambiguous ValueError (issue #135).\n        missing = [\n            doc\n            for doc in documents\n            if doc.embedding is None or len(doc.embedding) == 0\n        ]\n        if missing:\n            ids = [doc.id or \"<no id>\" for doc in missing]\n            raise ValueError(\n                f\"failed to embed {len(missing)} document(s): {ids}\"\n            )\n\n        # Batch the entire `documents` list into a single add_with_ids call.\n        # Per-document inserts would invalidate the SIMD-blocked cache\n        # between every doc.\n        vectors = np.asarray([doc.embedding for doc in documents], dtype=np.float32)\n        if vectors.ndim != 2:\n            raise ValueError(\n                f\"expected 2D embedding batch, got {vectors.ndim}D\"\n            )\n        if vectors.shape[1] != self.dimensions:\n            raise ValueError(\n                f\"embedding dim {vectors.shape[1]} does not match \"\n                f\"index dim {self.dimensions}\"\n            )\n        if not vectors.flags[\"C_CONTIGUOUS\"]:\n            vectors = np.ascontiguousarray(vectors)","sourceCodeStart":483,"sourceCodeEnd":519,"githubUrl":"https://github.com/RyanCodrai/turbovec/blob/ccab9f325e6ce2a270a87daf01ae4e443bcf2d49/turbovec-python/python/turbovec/agno.py#L483-L519","documentation":"insert() verifies every agno document received a non-empty embedding (the embedder was expected to fill doc.embedding). If any documents still have embedding None or empty, a ValueError lists how many and which ids failed, because the quantized index cannot ingest un-embedded documents.","triggerScenarios":"Calling insert() (or upsert(), which delegates to insert) with documents whose embedding is None/empty — usually when documents were constructed with embedding=None relying on the DB to embed, or the embedder silently returned empty vectors for some inputs.","commonSituations":"Embedding API rate limits or errors that yield no embedding for some docs; passing pre-baked Document objects without embeddings assuming auto-embed; embedder/model misconfiguration producing empty output for empty or oversized content.","solutions":["Embed documents before insert: docs = embedder.get_embedding_and_use(docs) (or the async equivalent).","Log/inspect the listed failing ids — check their content for empty strings or inputs the embedder rejects.","Add retry/error handling around the embedder call so transient API failures don't produce None embeddings.","Filter out or skip documents that fail embedding instead of passing them through to insert()."],"exampleFix":"// before\nvec_db.insert(documents)  # documents have embedding=None\n// after\nembedded = embedder.get_embedding_and_use(documents)\nvec_db.insert(embedded)","handlingStrategy":"validation","validationCode":"unembedded = [d for d in documents if not d.embedding]\nif unembedded:\n    documents = embedder.get_embedding_and_use(documents)","typeGuard":"def is_embedded(doc) -> bool:\n    return doc.embedding is not None and len(doc.embedding) > 0","tryCatchPattern":"try:\n    db.insert(content_hash, documents)\nexcept ValueError as e:\n    logger.error(\"embedding failed for some docs: %s\", e)\n    documents = [d for d in documents if is_embedded(d)]  # retry without failures","preventionTips":["Always run embedder.get_embedding_and_use(documents) before insert().","Add retry logic around embedder API calls for rate-limit/transient failures.","Validate document content (non-empty, within model limits) before embedding.","Log and drop documents that repeatedly fail to embed instead of failing the batch silently."],"tags":["embedding","agno","validation"],"backgroundTag":"empty-required-field","analyzedSha":"ccab9f325e6ce2a270a87daf01ae4e443bcf2d49","analyzedAt":"2026-09-06T08:39:18.516Z","contentChangedAt":"2026-09-06T08:39:18.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}