{"record":{"id":"8568c11018ae2780","repo":"RyanCodrai/turbovec","slug":"similarity-dot-product-produces-unbounded-raw-in","errorCode":null,"errorMessage":"similarity='dot_product' produces unbounded raw inner products, so relevance scores are not calibrated to [0, 1]; score_threshold filtering is only meaningful if your embeddings are unit-normalized upstream. Use similarity='cosine' (the default) for calibrated relevance scores.","messagePattern":"similarity='dot_product' produces unbounded raw inner products, so relevance scores are not calibrated to \\[0, 1\\]; score_threshold filtering is only meaningful if your embeddings are unit-normalized upstream\\. Use similarity='cosine' \\(the default\\) for calibrated relevance scores\\.","errorType":"console","errorClass":"UserWarning","httpStatus":null,"severity":"warning","filePath":"turbovec-python/python/turbovec/langchain.py","lineNumber":171,"sourceCode":"        return self._similarity\n\n    # ---- Relevance score normalization --------------------------------\n\n    def _select_relevance_score_fn(self) -> Callable[[float], float]:\n        # Under the default cosine mode both sides are unit vectors, so\n        # the engine's raw inner product is true cosine similarity in\n        # [-1, 1]; (sim + 1) / 2 maps it onto LangChain's [0, 1]\n        # relevance scale and the clamp only absorbs quantization noise.\n        if self._similarity == COSINE:\n            return lambda sim: max(0.0, min(1.0, (sim + 1.0) / 2.0))\n        # Under dot_product mode scores are raw inner products with no\n        # fixed range, so no mapping onto [0, 1] is meaningful. The same\n        # affine mapping is kept for continuity with earlier releases,\n        # but WITHOUT the clamp: clamping silently collapsed every raw\n        # score >= 1.0 onto exactly 1.0, which made score_threshold\n        # retrievers admit unrelated documents and suppressed the\n        # out-of-range warning VectorStore itself emits (issue #322).\n        warnings.warn(\n            \"similarity='dot_product' produces unbounded raw inner products, \"\n            \"so relevance scores are not calibrated to [0, 1]; \"\n            \"score_threshold filtering is only meaningful if your embeddings \"\n            \"are unit-normalized upstream. Use similarity='cosine' (the \"\n            \"default) for calibrated relevance scores.\",\n            UserWarning,\n            stacklevel=2,\n        )\n        return lambda sim: (sim + 1.0) / 2.0\n\n    # ---- Embedder-output validation -----------------------------------\n\n    @staticmethod\n    def _check_embedded_batch(vectors: np.ndarray, n_texts: int) -> None:\n        \"\"\"Validate the shape of an embedder's document-batch output.\n\n        Only 2D outputs are inspected here — any other ndim falls through\n        to ``_store_texts_and_vectors``, whose existing guard names the bad","sourceCodeStart":153,"sourceCodeEnd":189,"githubUrl":"https://github.com/RyanCodrai/turbovec/blob/ccab9f325e6ce2a270a87daf01ae4e443bcf2d49/turbovec-python/python/turbovec/langchain.py#L153-L189","documentation":"This is a Python UserWarning (not an exception) from the LangChain vector store's _select_relevance_score_fn: dot_product similarity yields unbounded raw inner products, so relevance scores are not calibrated to [0,1] and score_threshold filtering is unreliable unless embeddings are unit-normalized upstream. The affine mapping is kept for continuity, but the clamp was removed because it silently collapsed scores >= 1.0 onto 1.0 and let score_threshold retrievers admit unrelated documents (issue #322).","triggerScenarios":"Constructing the turbovec LangChain VectorStore with similarity='dot_product' and then using a score_threshold retriever (or any relevance-score mapping path).","commonSituations":"Switching similarity from the default 'cosine' to 'dot_product' for speed without normalizing embeddings; copying retriever configs that assume calibrated scores; upgrading to a release that removed the clamp.","solutions":["Use similarity='cosine' (the default) for calibrated [0,1] relevance scores","Unit-normalize embeddings before insertion/querying if you must use dot_product","Drop score_threshold filtering when using raw dot_product scores","Suppress or route the UserWarning only after confirming scores are bounded"],"exampleFix":"// before\nvs = Turbovec.from_documents(docs, emb, similarity=\"dot_product\")\nr = vs.as_retriever(search_kwargs={\"score_threshold\": 0.7})\n// after\nvs = Turbovec.from_documents(docs, emb)  # cosine, calibrated\nr = vs.as_retriever(search_kwargs={\"score_threshold\": 0.7})","handlingStrategy":"type-guard","validationCode":"import numpy as np\ndef embeddings_unit_norm(X):\n    n = np.linalg.norm(X, axis=-1)\n    return bool(np.all(np.abs(n - 1.0) < 1e-3))\n# only use dot_product + score_threshold if embeddings_unit_norm(X)","typeGuard":"import numpy as np\ndef is_unit_normalized(x) -> bool:\n    a = np.asarray(x)\n    return a.ndim >= 1 and bool(np.allclose(np.linalg.norm(a, axis=-1), 1.0, atol=1e-3))","tryCatchPattern":"import warnings\nwith warnings.catch_warnings(record=True) as caught:\n    warnings.simplefilter(\"always\")\n    scores = vs._select_relevance_score_fn(\"dot_product\")\nfor w in caught:\n    if \"dot_product\" in str(w.message):\n        logging.warning(\"unnormalized embeddings with dot_product: %s\", w.message)","preventionTips":["Normalize embeddings before indexing when using dot_product","Prefer the default 'cosine' similarity with score_threshold retrievers","Recheck retriever configs after upgrading releases that changed clamping behavior"],"tags":["python","langchain","warning","similarity"],"backgroundTag":"deprecated-api-usage","analyzedSha":"ccab9f325e6ce2a270a87daf01ae4e443bcf2d49","analyzedAt":"2026-09-06T08:39:18.516Z","contentChangedAt":"2026-09-06T08:39:18.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}