bytedance/deer-flow · error · ValueError

Duplicate fact ids are not allowed

Error message

Duplicate fact ids are not allowed

What it means

Raised by FileMemoryStorage.save() after acquiring the scope/file locks when two facts in the payload stringify to the same id (including the empty string, because missing ids become ''). Fact ids are the primary key used to diff old_ids vs new ids and compute deletes, so duplicates would make the upsert/delete delta ambiguous. The check is len(ids) != len(set(ids)).

Source

Thrown at backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/storage.py:1092

        lock_path = path.parent / ".memory.lock"
        notifications: list[RetrievalNotification] = []
        try:
            if not isinstance(memory_data, dict):
                raise ValueError("memory_data must be an object")
            if agent_name is not None and "facts" not in memory_data:
                raise ValueError("memory_data.facts is required for an agent full save")
            facts_raw = memory_data.get("facts", [])
            if not isinstance(facts_raw, list):
                raise ValueError("memory_data.facts must be a list")
            if any(not isinstance(fact, dict) for fact in facts_raw):
                raise ValueError("memory_data.facts must contain only fact objects")
            if agent_name is None and facts_raw:
                raise ValueError("agent_name is required to persist facts")
            with self._scope_lock(key), _process_file_lock(lock_path, float(getattr(self._config, "file_lock_timeout_seconds", 10))):
                self._recover_if_needed(path)
                ids = [str(fact.get("id") or "") for fact in facts_raw]
                if len(ids) != len(set(ids)):
                    raise ValueError("Duplicate fact ids are not allowed")
                old_ids = set(self._agent_entries(path, agent_name, user_id=user_id)) if agent_name is not None else set()
                summaries = None
                if agent_name is None:
                    summaries = {"user": memory_data.get("user", {}), "history": memory_data.get("history", {})}
                _, notifications = self._commit_changes_locked(
                    path,
                    user_id=user_id,
                    agent_name=agent_name,
                    upserts=copy.deepcopy(facts_raw),
                    deletes=sorted(old_ids - set(ids)),
                    summaries=summaries,
                    expected_revision=expected_revision,
                )
                document = self._read_document(path, agent_name, user_id=user_id)
                signature = self._scope_signature(path, agent_name)
                with self._cache_lock:
                    self._memory_cache[key] = (copy.deepcopy(document), signature)
        except MemoryRevisionConflict:

View on GitHub (pinned to 1dd6ba1acb)

Solutions

1. De-duplicate by id before saving, keeping the last/highest-revision entry per id.
2. Regenerate ids for facts missing them (fact['id'] = f"fact_{uuid.uuid4().hex}") so they no longer collide on ''.
3. If duplicates came from a merge, re-key imported facts with new ids and store the old id under a provenance field.

Example fix

# before
storage.save({"facts": facts}, agent_name=agent_name)  # facts has duplicate ids

# after
by_id = {}
for fact in facts:
    fact["id"] = str(fact.get("id") or f"fact_{uuid.uuid4().hex}")
    by_id[fact["id"]] = fact
storage.save({"facts": list(by_id.values())}, agent_name=agent_name)

ids = [str(f.get("id") or "") for f in facts]
if len(ids) != len(set(ids)):
    by_id = {}
    for f in sorted(facts, key=lambda x: x.get("revision") or 0):
        f.setdefault("id", f"fact_{uuid.uuid4().hex}")
        by_id[f["id"]] = f
    facts = list(by_id.values())
storage.save({"facts": facts}, agent_name=agent_name)

When it happens

Trigger: A full agent save whose facts array contains two entries with the same 'id', or two entries both missing 'id' (both normalize to '').

Common situations: LLM-generated fact batches reusing an id; merging two fact lists without re-keying; facts copied from another agent with ids regenerated only partially; two id-less facts in one payload.

Related errors

• retrieval fact.id must be a non-empty string
• retrieval fact.content must be a non-empty string
• unsupported FTS5 retrieval mode: {mode}
• retrieval category filter must be a string
• fact.category must be a string

AI-assisted analysis of bytedance/deer-flow@1dd6ba1acb (2026-08-14). Data as JSON: /api/errors/95837f2349ceb127. Report an issue: GitHub.