{"record":{"id":"95837f2349ceb127","repo":"bytedance/deer-flow","slug":"duplicate-fact-ids-are-not-allowed","errorCode":null,"errorMessage":"Duplicate fact ids are not allowed","messagePattern":"Duplicate fact ids are not allowed","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/storage.py","lineNumber":1092,"sourceCode":"        lock_path = path.parent / \".memory.lock\"\n        notifications: list[RetrievalNotification] = []\n        try:\n            if not isinstance(memory_data, dict):\n                raise ValueError(\"memory_data must be an object\")\n            if agent_name is not None and \"facts\" not in memory_data:\n                raise ValueError(\"memory_data.facts is required for an agent full save\")\n            facts_raw = memory_data.get(\"facts\", [])\n            if not isinstance(facts_raw, list):\n                raise ValueError(\"memory_data.facts must be a list\")\n            if any(not isinstance(fact, dict) for fact in facts_raw):\n                raise ValueError(\"memory_data.facts must contain only fact objects\")\n            if agent_name is None and facts_raw:\n                raise ValueError(\"agent_name is required to persist facts\")\n            with self._scope_lock(key), _process_file_lock(lock_path, float(getattr(self._config, \"file_lock_timeout_seconds\", 10))):\n                self._recover_if_needed(path)\n                ids = [str(fact.get(\"id\") or \"\") for fact in facts_raw]\n                if len(ids) != len(set(ids)):\n                    raise ValueError(\"Duplicate fact ids are not allowed\")\n                old_ids = set(self._agent_entries(path, agent_name, user_id=user_id)) if agent_name is not None else set()\n                summaries = None\n                if agent_name is None:\n                    summaries = {\"user\": memory_data.get(\"user\", {}), \"history\": memory_data.get(\"history\", {})}\n                _, notifications = self._commit_changes_locked(\n                    path,\n                    user_id=user_id,\n                    agent_name=agent_name,\n                    upserts=copy.deepcopy(facts_raw),\n                    deletes=sorted(old_ids - set(ids)),\n                    summaries=summaries,\n                    expected_revision=expected_revision,\n                )\n                document = self._read_document(path, agent_name, user_id=user_id)\n                signature = self._scope_signature(path, agent_name)\n                with self._cache_lock:\n                    self._memory_cache[key] = (copy.deepcopy(document), signature)\n        except MemoryRevisionConflict:","sourceCodeStart":1074,"sourceCodeEnd":1110,"githubUrl":"https://github.com/bytedance/deer-flow/blob/1dd6ba1acb03700589994b0366c5d1c7d05e2eff/backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/storage.py#L1074-L1110","documentation":"Raised by FileMemoryStorage.save() after acquiring the scope/file locks when two facts in the payload stringify to the same id (including the empty string, because missing ids become ''). Fact ids are the primary key used to diff old_ids vs new ids and compute deletes, so duplicates would make the upsert/delete delta ambiguous. The check is len(ids) != len(set(ids)).","triggerScenarios":"A full agent save whose facts array contains two entries with the same 'id', or two entries both missing 'id' (both normalize to '').","commonSituations":"LLM-generated fact batches reusing an id; merging two fact lists without re-keying; facts copied from another agent with ids regenerated only partially; two id-less facts in one payload.","solutions":["De-duplicate by id before saving, keeping the last/highest-revision entry per id.","Regenerate ids for facts missing them (fact['id'] = f\"fact_{uuid.uuid4().hex}\") so they no longer collide on ''.","If duplicates came from a merge, re-key imported facts with new ids and store the old id under a provenance field."],"exampleFix":"# before\nstorage.save({\"facts\": facts}, agent_name=agent_name)  # facts has duplicate ids\n\n# after\nby_id = {}\nfor fact in facts:\n    fact[\"id\"] = str(fact.get(\"id\") or f\"fact_{uuid.uuid4().hex}\")\n    by_id[fact[\"id\"]] = fact\nstorage.save({\"facts\": list(by_id.values())}, agent_name=agent_name)","handlingStrategy":"validation","validationCode":"ids = [str(f.get(\"id\") or \"\") for f in facts]\nif len(ids) != len(set(ids)):\n    by_id = {}\n    for f in sorted(facts, key=lambda x: x.get(\"revision\") or 0):\n        f.setdefault(\"id\", f\"fact_{uuid.uuid4().hex}\")\n        by_id[f[\"id\"]] = f\n    facts = list(by_id.values())\nstorage.save({\"facts\": facts}, agent_name=agent_name)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always assign an id when creating facts; never rely on id-less entries surviving a batch save.","De-duplicate merged fact lists by id before saving.","Regenerate ids when copying facts between agents."],"tags":["memory","validation","duplicate-key","deermem"],"backgroundTag":null,"analyzedSha":"1dd6ba1acb03700589994b0366c5d1c7d05e2eff","analyzedAt":"2026-08-14T21:20:34.804Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}