{"record":{"id":"cfe5c010940d541e","repo":"ScrapeGraphAI/Scrapegraph-ai","slug":"if-set-timeout-value-for-scrolling-scraper-must-b","errorCode":null,"errorMessage":"If set, timeout value for scrolling scraper must be greater than 0.","messagePattern":"If set, timeout value for scrolling scraper must be greater than 0\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scrapegraphai/docloaders/chromium.py","lineNumber":235,"sourceCode":"        Less than this and we don't scroll enough to see any content change.\n        - sleep (int): The number of seconds to sleep after each scroll, to allow the page to load.\n        Defaults to 2. Must be greater than 0.\n\n        Returns:\n            str: The scraped HTML content\n\n        Raises:\n        - ValueError: If the timeout value is less than or equal to 0.\n        - ValueError: If the sleep value is less than or equal to 0.\n        - ValueError: If the scroll value is less than 5000.\n        \"\"\"\n        # NB: I have tested using scrollHeight to determine when to stop scrolling\n        # but it doesn't always work as expected. The page height doesn't change on some sites like\n        # https://www.steelwood.amsterdam/. The site deos not scroll to the bottom.\n        # In my browser I can scroll vertically but in Chromium it scrolls horizontally?!?\n\n        if timeout and timeout <= 0:\n            raise ValueError(\n                \"If set, timeout value for scrolling scraper must be greater than 0.\"\n            )\n\n        if sleep <= 0:\n            raise ValueError(\n                \"Sleep for scrolling scraper value must be greater than 0.\"\n            )\n\n        if scroll < 5000:\n            raise ValueError(\n                \"Scroll value for scrolling scraper must be greater than or equal to 5000.\"\n            )\n\n        import time\n\n        from playwright.async_api import async_playwright\n        from undetected_playwright import Malenia\n","sourceCodeStart":217,"sourceCodeEnd":253,"githubUrl":"https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/532dfffbf6ee823a6c9cf8cfedc24a93bf026780/scrapegraphai/docloaders/chromium.py#L217-L253","documentation":"create_batch in utils/batch_api.py rejects any request list longer than MAX_REQUESTS_PER_BATCH (the OpenAI Batch API hard limit of 50,000 requests). The ValueError tells you to split the workload, because the API would reject an oversized JSONL file anyway.","triggerScenarios":"Calling create_batch with len(requests) > MAX_REQUESTS_PER_BATCH, e.g. batch-evaluating a large dataset or backfilling embeddings for hundreds of thousands of rows in a single call.","commonSituations":"Bulk jobs that grew past 50k rows after a data expansion; migrating a loop-based script to the batch API without chunking; concatenating multiple datasets into one batch request.","solutions":["Chunk the request list into batches of at most MAX_REQUESTS_PER_BATCH and call create_batch per chunk, collecting the returned batch IDs.","Track all returned batch IDs and poll each with retrieve_batch until all complete.","For very large jobs, also respect file-size limits by splitting further (e.g. 20-30k requests per batch)."],"exampleFix":"# before\nbatch_id = create_batch(requests)  # 120k requests -> ValueError\n\n# after\nBATCH = 50_000\nbatch_ids = [create_batch(requests[i:i+BATCH]) for i in range(0, len(requests), BATCH)]","handlingStrategy":"validation","validationCode":"MAX = 50_000  # MAX_REQUESTS_PER_BATCH\nif len(requests) > MAX:\n    batches = [requests[i:i+MAX] for i in range(0, len(requests), MAX)]\nelse:\n    batches = [requests]\nbatch_ids = [create_batch(b) for b in batches]","typeGuard":"def fits_single_batch(requests: list) -> bool:\n    return len(requests) <= 50_000","tryCatchPattern":"try:\n    batch_id = create_batch(requests)\nexcept ValueError as e:\n    if \"exceeds the maximum\" in str(e):\n        chunks = [requests[i:i+50_000] for i in range(0, len(requests), 50_000)]\n        batch_ids = [create_batch(c) for c in chunks]","preventionTips":["Chunk requests before calling create_batch; never assume arbitrary list sizes are accepted.","Track multiple batch IDs and poll each with retrieve_batch.","Split large jobs further to also stay under file-size limits."],"tags":["batch-api","rate-limits","chunking","openai"],"backgroundTag":"batch-size-limit-exceeded","analyzedSha":"532dfffbf6ee823a6c9cf8cfedc24a93bf026780","analyzedAt":"2026-08-28T15:19:38.821Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}