bytedance/deer-flow · warning · ValueError

Duplicate fact

Error message

Duplicate fact

What it means

_raise_if_duplicate_fact_content() rejects a candidate fact whose normalized content already exists in the current snapshot; _fact_content_key() normalizes the content string so trivial differences (whitespace/case, per its normalization) still collide. It is designed to be re-run inside the read-check-write critical section on every revision-conflict retry, so two concurrent creators of the same content cannot both store it.

Source

Thrown at backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/updater.py:470

        return None
    stripped = content.strip()
    if not stripped:
        return None
    return stripped.casefold()


def _raise_if_duplicate_fact_content(memory_data: dict[str, Any], content_key: str | None) -> None:
    """Reject a candidate fact whose normalized content already exists.

    Callers must invoke this against the freshest snapshot available inside
    their read-check-write critical section (i.e. on every revision-conflict
    retry), so two concurrent creators of the same content cannot both pass
    the check and store duplicate facts."""
    if content_key is None:
        return
    for fact in memory_data.get("facts", []):
        if isinstance(fact, dict) and _fact_content_key(fact.get("content")) == content_key:
            raise ValueError("Duplicate fact")


# ── Staleness review helpers ──────────────────────────────────────────────


def _parse_fact_datetime(raw: str) -> datetime | None:
    """Parse an ISO-8601 datetime string from a fact's createdAt field.

    Returns ``None`` on any parse failure so callers can safely skip malformed facts.
    """
    if not raw:
        return None
    try:
        result = datetime.fromisoformat(raw)
        # Naive datetimes (no tzinfo) would cause TypeError when compared
        # with the timezone-aware cutoff.  Assume UTC for safety.
        if result.tzinfo is None:
            result = result.replace(tzinfo=UTC)

View on GitHub (pinned to 1dd6ba1acb)

Solutions

1. Treat the error as 'already known': catch ValueError, check that str(e) == 'Duplicate fact', and return the existing fact (lookup via list_facts with the content key) instead of failing the request.
2. Before creating, query for the normalized content and update the existing fact's revision/metadata instead of inserting.
3. Debounce or dedupe extraction runs so identical extractions within a window collapse into one call.

Example fix

# before
updater.create_memory_fact(agent_name=a, content="User lives in Berlin", confidence=0.9)

# after
try:
    updater.create_memory_fact(agent_name=a, content="User lives in Berlin", confidence=0.9)
except ValueError as exc:
    if str(exc) != "Duplicate fact":
        raise
    # already stored; refresh it instead
    facts = storage.list_facts(agent_name=a, filters={})
    existing = next(f for f in facts if normalize(f["content"]) == normalize("User lives in Berlin"))
    storage.upsert_fact({**existing, "confidence": 0.9}, agent_name=a)

existing = [f for f in storage.list_facts(agent_name=agent_name) if _fact_content_key(f.get("content")) == _fact_content_key(new_content)]
if existing:
    return existing[0]  # skip create, update instead

try:
    updater.create_memory_fact(agent_name=a, content=content, confidence=c)
except ValueError as exc:
    if str(exc) != "Duplicate fact":
        raise
    logger.info("fact already present; updating instead")
    # fall through to an update path on the existing fact

When it happens

Trigger: create_memory_fact called with content the agent already stored (e.g. the same preference re-extracted on a later turn); concurrent extraction of the same fact by two threads; content differing only in ways _fact_content_key normalizes away.

Common situations: Memory extraction running on every turn without dedup awareness; re-importing an exported memory file; retrying a user request that re-creates the same fact; users restating the same information phrased identically.

Related errors

• Missing or empty 'messages' key in {path}
• chat prompt template not found: {name} (searched: {searched}
• guaranteed_categories must be an iterable of strings, not a
• memory update queue is full (depth {len(self._items)} >= {ma
• retrieval scope userId must be a string or null

AI-assisted analysis of bytedance/deer-flow@1dd6ba1acb (2026-08-14). Data as JSON: /api/errors/0dffd5b2d22d74b7. Report an issue: GitHub.