bytedance/deer-flow · critical · MemoryStorageCorruption

Existing migration backup {backup_path} differs from source

Error message

Existing migration backup {backup_path} differs from source {source_path}; the original backup was kept and migration was stopped

What it means

During migration, deermem writes an immutable .v1.bak backup of the source JSON next to it. If a backup already exists but its bytes differ from the current source file, the migration halts (MemoryStorageCorruption) and the original backup is preserved. The check guarantees the backup always reflects the exact pre-migration state, so a mismatch means a previous migration attempt or external edit intervened.

Source

Thrown at backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/storage.py:146


def _file_signature(path: Path) -> tuple[int, int] | None:
    """Use nanosecond mtime plus size so cache validation is not mtime-only."""
    try:
        stat = path.stat()
        return (stat.st_mtime_ns, stat.st_size)
    except OSError:
        return None


def _ensure_migration_backup(source_path: Path) -> Path:
    """Durably preserve one immutable pre-migration JSON source beside it."""
    backup_path = source_path.with_name(f"{source_path.name}.v1.bak")
    try:
        source_bytes = source_path.read_bytes()
        if backup_path.exists():
            if backup_path.read_bytes() != source_bytes:
                raise MemoryStorageCorruption(f"Existing migration backup {backup_path} differs from source {source_path}; the original backup was kept and migration was stopped")
            return backup_path
        _atomic_write(backup_path, source_bytes)
        return backup_path
    except MemoryStorageCorruption:
        raise
    except OSError as exc:
        raise OSError(f"Failed to create durable migration backup {backup_path}: {exc}") from exc


def _normalize_category(fact: dict[str, Any]) -> None:
    raw_category = fact.get("category", "context")
    if not isinstance(raw_category, str):
        raise ValueError("fact.category must be a string")
    category = raw_category or "context"
    if category not in CORE_CATEGORIES:
        fact.setdefault("categoryExtension", category)
        fact["category"] = "other"

View on GitHub (pinned to 1dd6ba1acb)

Solutions

  1. Compare memory.json with memory.json.v1.bak; if memory.json is the authoritative current state, delete the stale .v1.bak and retry so a fresh backup is written.
  2. If the .v1.bak is the true pre-migration snapshot you must keep, restore memory.json from it byte-for-byte before retrying.
  3. Ensure only one process migrates a given memory directory at a time (single instance, or stop the service before upgrading).
Defensive patterns

Strategy: try-catch

Validate before calling

from pathlib import Path

def backup_consistent(source: Path) -> bool:
    bak = source.with_name(source.name + ".v1.bak")
    return not bak.exists() or bak.read_bytes() == source.read_bytes()

Try / catch

try:
    store.migrate()
except MemoryStorageCorruption as exc:
    if "differs from source" in str(exc):
        # decide which of memory.json / memory.json.v1.bak is authoritative,
        # reconcile by hand, delete the stale one, retry
        raise

Prevention

When it happens

Trigger: A first migration created memory.json.v1.bak; afterwards memory.json changed (second legacy write, partial migration, manual edit) and migration runs again, so the bytes no longer match. Also triggered by restoring memory.json from a different backup while keeping the old .v1.bak.

Common situations: Crashed or killed process midway through migration leaving divergent files; running two DeerFlow instances against one memory directory; sync tools (Dropbox/Nextcloud) restoring conflicting file versions.

Related errors

AI-assisted analysis of bytedance/deer-flow@1dd6ba1acb (2026-08-14). Data as JSON: /api/errors/43af5d0fd3666bf9. Report an issue: GitHub.