ScrapeGraphAI/Scrapegraph-ai · error · ValueError

If set, timeout value for scrolling scraper must be greater…

Error message

If set, timeout value for scrolling scraper must be greater than 0.

What it means

create_batch in utils/batch_api.py rejects any request list longer than MAX_REQUESTS_PER_BATCH (the OpenAI Batch API hard limit of 50,000 requests). The ValueError tells you to split the workload, because the API would reject an oversized JSONL file anyway.

Solutions

1. Chunk the request list into batches of at most MAX_REQUESTS_PER_BATCH and call create_batch per chunk, collecting the returned batch IDs.
2. Track all returned batch IDs and poll each with retrieve_batch until all complete.
3. For very large jobs, also respect file-size limits by splitting further (e.g. 20-30k requests per batch).

Example fix

# before
batch_id = create_batch(requests)  # 120k requests -> ValueError

# after
BATCH = 50_000
batch_ids = [create_batch(requests[i:i+BATCH]) for i in range(0, len(requests), BATCH)]

MAX = 50_000  # MAX_REQUESTS_PER_BATCH
if len(requests) > MAX:
    batches = [requests[i:i+MAX] for i in range(0, len(requests), MAX)]
else:
    batches = [requests]
batch_ids = [create_batch(b) for b in batches]

def fits_single_batch(requests: list) -> bool:
    return len(requests) <= 50_000

try:
    batch_id = create_batch(requests)
except ValueError as e:
    if "exceeds the maximum" in str(e):
        chunks = [requests[i:i+50_000] for i in range(0, len(requests), 50_000)]
        batch_ids = [create_batch(c) for c in chunks]

When it happens

Trigger: Calling create_batch with len(requests) > MAX_REQUESTS_PER_BATCH, e.g. batch-evaluating a large dataset or backfilling embeddings for hundreds of thousands of rows in a single call.

Common situations: Bulk jobs that grew past 50k rows after a data expansion; migrating a loop-based script to the batch API without chunking; concatenating multiple datasets into one batch request.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

AI-assisted analysis of ScrapeGraphAI/Scrapegraph-ai@532dfffbf6 (2026-08-28). Data as JSON: /api/errors/cfe5c010940d541e. Report an issue: GitHub.

Appendix: source

Thrown at scrapegraphai/docloaders/chromium.py:235

        Less than this and we don't scroll enough to see any content change.
        - sleep (int): The number of seconds to sleep after each scroll, to allow the page to load.
        Defaults to 2. Must be greater than 0.

        Returns:
            str: The scraped HTML content

        Raises:
        - ValueError: If the timeout value is less than or equal to 0.
        - ValueError: If the sleep value is less than or equal to 0.
        - ValueError: If the scroll value is less than 5000.
        """
        # NB: I have tested using scrollHeight to determine when to stop scrolling
        # but it doesn't always work as expected. The page height doesn't change on some sites like
        # https://www.steelwood.amsterdam/. The site deos not scroll to the bottom.
        # In my browser I can scroll vertically but in Chromium it scrolls horizontally?!?

        if timeout and timeout <= 0:
            raise ValueError(
                "If set, timeout value for scrolling scraper must be greater than 0."
            )

        if sleep <= 0:
            raise ValueError(
                "Sleep for scrolling scraper value must be greater than 0."
            )

        if scroll < 5000:
            raise ValueError(
                "Scroll value for scrolling scraper must be greater than or equal to 5000."
            )

        import time

        from playwright.async_api import async_playwright
        from undetected_playwright import Malenia

View on GitHub (pinned to 532dfffbf6)