D4Vinci/Scrapling · error · ValueError

Checkpoints interval must be equal or greater than 0.

Error message

Checkpoints interval must be equal or greater than 0.

What it means

CheckpointManager rejects negative intervals: a checkpoint every N seconds makes no sense for N < 0, so __init__ raises ValueError. Zero is allowed (checkpoint on every save opportunity), as is any positive int/float.

Source

Thrown at scrapling/spiders/checkpoint.py:36

    requests: List["Request"] = field(default_factory=list)
    seen: Set[bytes] = field(default_factory=set)


class CheckpointManager:
    """Manages saving and loading checkpoint state to/from disk."""

    CHECKPOINT_FILE = "checkpoint.pkl"

    def __init__(self, crawldir: str | Path | AsyncPath, interval: float = 300.0):
        self.crawldir = AsyncPath(crawldir)
        self._checkpoint_path = self.crawldir / self.CHECKPOINT_FILE
        self.interval = interval
        if not isinstance(interval, (int, float)):
            raise TypeError("Checkpoints interval must be integer or float.")
        else:
            if interval < 0:
                raise ValueError("Checkpoints interval must be equal or greater than 0.")

    async def has_checkpoint(self) -> bool:
        """Check if a checkpoint exists."""
        return await self._checkpoint_path.exists()

    async def save(self, data: CheckpointData) -> None:
        """Save checkpoint data to disk atomically."""
        await self.crawldir.mkdir(parents=True, exist_ok=True)

        temp_path = self._checkpoint_path.with_suffix(".tmp")

        try:
            serialized = pickle.dumps(data, protocol=pickle.HIGHEST_PROTOCOL)
            async with await anyio.open_file(temp_path, "wb") as f:
                await f.write(serialized)

            await temp_path.replace(self._checkpoint_path)

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Clamp the computed value: interval=max(0, computed_interval).
2. If -1 means 'disable checkpoints' in your config, branch and skip creating the manager or use interval=0 deliberately.
3. Audit the arithmetic that produced the interval — a negative value usually indicates a deadline already passed.

Example fix

# before
CheckpointManager(crawldir, interval=remaining_seconds)  # ValueError if negative

# after
CheckpointManager(crawldir, interval=max(0.0, remaining_seconds))

interval = float(cfg.get('interval', 300))
interval = max(0.0, interval)  # zero is allowed, negatives are not
CheckpointManager(crawldir, interval=interval)

def is_non_negative_number(value) -> bool:
    return isinstance(value, (int, float)) and not isinstance(value, bool) and value >= 0

When it happens

Trigger: CheckpointManager(crawldir, interval=-300) or interval=-0.5, often from arithmetic that produced a negative number (e.g. subtracting a larger configured delay from a budget).

Common situations: Computing interval as a difference (deadline - elapsed) that went negative; a sentinel value like -1 meaning 'disabled' leaking from config into the manager.

Related errors

• Checkpoints interval must be integer or float.
• Cannot use 'proxy_rotator' together with 'proxy' or 'proxies
• Unknown parser argument: "{key}"; maybe you meant {cls.parse
• You must pass a keyword to configure, current keywords: {cls
• Invalid proxy string!

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/f7381c4f1bb90788. Report an issue: GitHub.