D4Vinci/Scrapling · error · TypeError

Checkpoints interval must be integer or float.

Error message

Checkpoints interval must be integer or float.

What it means

CheckpointManager validates its checkpointing interval in __init__ and requires an int or float. Passing anything else (str, None, bool excluded types, Decimal) raises TypeError immediately at construction, before any crawling starts.

Solutions

1. Cast at the call site: CheckpointManager(crawldir, interval=float(cfg['interval'])).
2. If the value is a timedelta, use interval=td.total_seconds().
3. Validate config early with a schema/type check so bad values surface at startup with context.

Example fix

# before
CheckpointManager(crawldir, interval=os.environ['CHECKPOINT_INTERVAL'])  # TypeError

# after
CheckpointManager(crawldir, interval=float(os.environ['CHECKPOINT_INTERVAL']))

def to_interval_seconds(value) -> float:
    if isinstance(value, str):
        value = float(value)
    if not isinstance(value, (int, float)) or isinstance(value, bool):
        raise TypeError(f'interval must be int/float, got {type(value).__name__}')
    return float(value)

CheckpointManager(crawldir, interval=to_interval_seconds(cfg['interval']))

def is_interval(value) -> bool:
    return isinstance(value, (int, float)) and not isinstance(value, bool)

When it happens

Trigger: CheckpointManager(crawldir, interval='300') — a string read from CLI args, environment variables, or a YAML/JSON config that wasn't cast to a number.

Common situations: Loading interval from settings: os.environ['INTERVAL'] or a config file yields str; passing a datetime.timedelta instead of seconds.

Related errors

• Checkpoints interval must be equal or greater than 0.
• Argument `selector_config` must be a dictionary.
• Argument `selector_config` must be a dictionary.
• Argument with type " " is not accepted, please read the…
• `autothrottle_max_delay` can't be lower than…

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/17e5254f571fedff. Report an issue: GitHub.

Appendix: source

Thrown at scrapling/spiders/checkpoint.py:33

@dataclass
class CheckpointData:
    """Container for checkpoint state."""

    requests: List["Request"] = field(default_factory=list)
    seen: Set[bytes] = field(default_factory=set)


class CheckpointManager:
    """Manages saving and loading checkpoint state to/from disk."""

    CHECKPOINT_FILE = "checkpoint.pkl"

    def __init__(self, crawldir: str | Path | AsyncPath, interval: float = 300.0):
        self.crawldir = AsyncPath(crawldir)
        self._checkpoint_path = self.crawldir / self.CHECKPOINT_FILE
        self.interval = interval
        if not isinstance(interval, (int, float)):
            raise TypeError("Checkpoints interval must be integer or float.")
        else:
            if interval < 0:
                raise ValueError("Checkpoints interval must be equal or greater than 0.")

    async def has_checkpoint(self) -> bool:
        """Check if a checkpoint exists."""
        return await self._checkpoint_path.exists()

    async def save(self, data: CheckpointData) -> None:
        """Save checkpoint data to disk atomically."""
        await self.crawldir.mkdir(parents=True, exist_ok=True)

        temp_path = self._checkpoint_path.with_suffix(".tmp")

        try:
            serialized = pickle.dumps(data, protocol=pickle.HIGHEST_PROTOCOL)
            async with await anyio.open_file(temp_path, "wb") as f:
                await f.write(serialized)

View on GitHub (pinned to 5d213a2d47)