What causes "Storage system must implement `retrieve` method" in D4Vinci/Scrapling?

Implement `def retrieve(self, identifier: str) -> Optional[Dict]` in your subclass, returning the saved property dict or `None` when not found.

D4Vinci/Scrapling · error · NotImplementedError

Storage system must implement `retrieve` method

Error message

Storage system must implement `retrieve` method

What it means

Raised by the abstract `retrieve` method of `StorageSystemMixin` in scrapling/core/storage.py. `retrieve(identifier)` is what adaptive fetching calls to look up an element's previously saved properties (via `element.retrieve()` / auto-retrieve). Like its `save` sibling, the ABC raises `NotImplementedError` when a custom storage backend fails to override it. The bundled `SQLiteStorageSystem` implements lookup against its SQLite file, so this error only appears with user-defined storage classes.

Source

Thrown at scrapling/core/storage.py:59

    @abstractmethod
    def save(self, element: HtmlElement, identifier: str) -> None:
        """Saves the element's unique properties to the storage for retrieval and relocation later

        :param element: The element itself which we want to save to storage.
        :param identifier: This is the identifier that will be used to retrieve the element later from the storage. See
            the docs for more info.
        """
        raise NotImplementedError("Storage system must implement `save` method")

    @abstractmethod
    def retrieve(self, identifier: str) -> Optional[Dict]:
        """Using the identifier, we search the storage and return the unique properties of the element

        :param identifier: This is the identifier that will be used to retrieve the element from the storage. See
            the docs for more info.
        :return: A dictionary of the unique properties
        """
        raise NotImplementedError("Storage system must implement `retrieve` method")

    @staticmethod
    @lru_cache(128, typed=True)
    def _get_hash(identifier: str) -> str:
        """If you want to hash identifier in your storage system, use this safer"""
        _identifier = identifier.lower().strip()
        # Hash functions have to take bytes
        _identifier_bytes = _identifier.encode("utf-8")

        hash_value = sha256(_identifier_bytes).hexdigest()
        return f"{hash_value}_{len(_identifier_bytes)}"  # Length to reduce collision chance


@lru_cache(1, typed=True)
class SQLiteStorageSystem(StorageSystemMixin):
    """The recommended system to use, it's race condition safe and thread safe.
    Mainly built, so the library can run in threaded frameworks like scrapy or threaded tools
    > It's optimized for threaded applications, but running it without threads shouldn't make it slow."""

View on GitHub (pinned to 5d213a2d47)

Solutions

Implement `def retrieve(self, identifier: str) -> Optional[Dict]` in your subclass, returning the saved property dict or `None` when not found.
Mirror the reference implementation in `SQLiteStorageSystem.retrieve` for the expected dict shape (`_StorageTools.element_to_dict` output).
Guard with `inspect.isabstract()` at import time so a half-finished backend fails fast.
Use the built-in SQLite backend if you don't actually need a custom store.

Example fix

// before
class MyStorage(StorageSystemMixin):
    def save(self, element, identifier): ...
    # retrieve missing

// after
class MyStorage(StorageSystemMixin):
    def save(self, element, identifier): ...

    def retrieve(self, identifier) -> Optional[Dict]:
        row = self._db.get(self._get_hash(identifier))
        return row  # dict of unique properties, or None

Defensive patterns

Strategy: validation

Validate before calling

def assert_storage_retrieve(storage) -> None:
    import inspect
    m = type(storage).retrieve
    assert getattr(m, '__isabstractmethod__', False) is False or storage.__class__ is not None, 'retrieve not overridden'
    # strongest check: no abstract methods left on the concrete class
    assert not type(storage).__abstractmethods__, f"abstract methods left: {type(storage).__abstractmethods__}"

Type guard

def is_concrete_storage(s) -> bool:
    cls = type(s)
    return not getattr(cls, '__abstractmethods__', None) and callable(getattr(cls, 'retrieve', None)) and not getattr(cls.retrieve, '__isabstractmethod__', False)

Try / catch

try:
    props = element.retrieve(identifier='x')
except NotImplementedError:
    log.error('storage backend lacks retrieve(); implement it before adaptive lookup')
    raise

Prevention

Implement both save and retrieve together — the interface is symmetric.
Return Optional[Dict]: None when not found instead of raising.
Unit-test retrieve with a saved identifier right after save in your backend's test suite.

When it happens

Trigger: A custom `StorageSystemMixin` subclass that implements `save` but not `retrieve`, followed by any adaptive retrieval call: `page.css('h1').relocate()` or constructing a Selector with `adaptive=True` / `storage=MyStorage()` so retrieval looks up a saved identifier.

Common situations: Porting a storage backend from an older Scrapling version where the interface had fewer methods; implementing only the write path during development and forgetting the read path; typo'd method name such as `get` or `load` instead of `retrieve`.

Related errors

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/fec47366b2fb7e28. Report an issue: GitHub.