{"record":{"id":"2b59be712459ded6","repo":"crewAIInc/crewAI","slug":"polling-timed-out-before-job-completed","errorCode":null,"errorMessage":"Polling timed out before job completed.","messagePattern":"Polling timed out before job completed\\.","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/tools/brightdata_tool/brightdata_dataset.py","lineNumber":535,"sourceCode":"\n                async with session.get(\n                    f\"{BRIGHTDATA_API_URL}/datasets/v3/progress/{snapshot_id}\",\n                    headers=headers,\n                ) as status_response:\n                    if status_response.status != 200:\n                        raise BrightDataDatasetToolException(\n                            f\"Status check failed: {await status_response.text()}\",\n                            status_response.status,\n                        )\n                    status_data = await status_response.json()\n                    if status_data.get(\"status\") == \"ready\":\n                        break\n                    if status_data.get(\"status\") == \"error\":\n                        raise BrightDataDatasetToolException(\n                            f\"Job failed: {status_data}\", 0\n                        )\n            else:\n                raise TimeoutError(\"Polling timed out before job completed.\")\n\n            async with session.get(\n                f\"{BRIGHTDATA_API_URL}/datasets/v3/snapshot/{snapshot_id}\",\n                params={\"format\": output_format},\n                headers=headers,\n            ) as snapshot_response:\n                if snapshot_response.status != 200:\n                    raise BrightDataDatasetToolException(\n                        f\"Result fetch failed: {await snapshot_response.text()}\",\n                        snapshot_response.status,\n                    )\n\n                return await snapshot_response.text()\n\n    def _run(\n        self,\n        url: str | None = None,\n        dataset_type: str | None = None,","sourceCodeStart":517,"sourceCodeEnd":553,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/tools/brightdata_tool/brightdata_dataset.py#L517-L553","documentation":"TimeoutError raised by the dataset tool's while/else construct: the polling loop ran for the full `timeout` seconds without the snapshot reaching 'ready' (and without an explicit error status). Note the loop sleeps polling_interval before each check, so elapsed time accumulates in discrete chunks.","triggerScenarios":"Calling get_dataset_data with default timeout while the dataset legitimately takes longer (large Amazon/map datasets can take minutes); a stuck snapshot that never transitions from 'running'/'pending' to ready or error.","commonSituations":"Deep-scrape dataset types under load, timeout parameter not scaled with input size, short defaults in library versions vs slower Bright Data processing.","solutions":["Increase the timeout argument (and optionally polling_interval) when requesting large or slow datasets.","Check the snapshot in the Bright Data dashboard — if stuck, discard and re-trigger.","Catch TimeoutError at the call site (the tool re-raises it after wrapping) and retry or degrade gracefully.","Split very large requests into smaller batches so each snapshot finishes within timeout."],"exampleFix":"# before\nresult = tool.run(url=url, dataset_type='amazon_product')  # default timeout\n\n# after\nresult = tool.run(url=url, dataset_type='amazon_product', timeout=600, polling_interval=10)","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    result = tool.run(url=url, dataset_type=dataset_type, timeout=300)\nexcept TimeoutError:\n    # snapshot may still complete later; retry once with a longer budget\n    result = tool.run(url=url, dataset_type=dataset_type, timeout=900, polling_interval=15)","preventionTips":["Scale the timeout argument to dataset size — deep scrapes need minutes, not seconds.","Catch TimeoutError separately from BrightDataDatasetToolException at the call site.","Batch large inputs into multiple smaller trigger calls so each snapshot finishes faster."],"tags":["bright-data","timeout","polling","dataset"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}