D4Vinci/Scrapling · error · TypeError

Checkpoints interval must be integer or float.

Error message

Checkpoints interval must be integer or float.

What it means

CheckpointManager validates its checkpointing interval in __init__ and requires an int or float. Passing anything else (str, None, bool excluded types, Decimal) raises TypeError immediately at construction, before any crawling starts.

Source

Thrown at scrapling/spiders/checkpoint.py:33

@dataclass
class CheckpointData:
    """Container for checkpoint state."""

    requests: List["Request"] = field(default_factory=list)
    seen: Set[bytes] = field(default_factory=set)


class CheckpointManager:
    """Manages saving and loading checkpoint state to/from disk."""

    CHECKPOINT_FILE = "checkpoint.pkl"

    def __init__(self, crawldir: str | Path | AsyncPath, interval: float = 300.0):
        self.crawldir = AsyncPath(crawldir)
        self._checkpoint_path = self.crawldir / self.CHECKPOINT_FILE
        self.interval = interval
        if not isinstance(interval, (int, float)):
            raise TypeError("Checkpoints interval must be integer or float.")
        else:
            if interval < 0:
                raise ValueError("Checkpoints interval must be equal or greater than 0.")

    async def has_checkpoint(self) -> bool:
        """Check if a checkpoint exists."""
        return await self._checkpoint_path.exists()

    async def save(self, data: CheckpointData) -> None:
        """Save checkpoint data to disk atomically."""
        await self.crawldir.mkdir(parents=True, exist_ok=True)

        temp_path = self._checkpoint_path.with_suffix(".tmp")

        try:
            serialized = pickle.dumps(data, protocol=pickle.HIGHEST_PROTOCOL)
            async with await anyio.open_file(temp_path, "wb") as f:
                await f.write(serialized)

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Cast at the call site: CheckpointManager(crawldir, interval=float(cfg['interval'])).
2. If the value is a timedelta, use interval=td.total_seconds().
3. Validate config early with a schema/type check so bad values surface at startup with context.

Example fix

# before
CheckpointManager(crawldir, interval=os.environ['CHECKPOINT_INTERVAL'])  # TypeError

# after
CheckpointManager(crawldir, interval=float(os.environ['CHECKPOINT_INTERVAL']))

def to_interval_seconds(value) -> float:
    if isinstance(value, str):
        value = float(value)
    if not isinstance(value, (int, float)) or isinstance(value, bool):
        raise TypeError(f'interval must be int/float, got {type(value).__name__}')
    return float(value)

CheckpointManager(crawldir, interval=to_interval_seconds(cfg['interval']))

def is_interval(value) -> bool:
    return isinstance(value, (int, float)) and not isinstance(value, bool)

When it happens

Trigger: CheckpointManager(crawldir, interval='300') — a string read from CLI args, environment variables, or a YAML/JSON config that wasn't cast to a number.

Common situations: Loading interval from settings: os.environ['INTERVAL'] or a config file yields str; passing a datetime.timedelta instead of seconds.

Related errors

• Checkpoints interval must be equal or greater than 0.
• Cannot use 'proxy_rotator' together with 'proxy' or 'proxies
• Response meta should be dictionary but got {type(meta).__nam
• Unknown parser argument: "{key}"; maybe you meant {cls.parse
• You must pass a keyword to configure, current keywords: {cls

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/17e5254f571fedff. Report an issue: GitHub.