crewAIInc/crewAI · error · TimeoutError

Polling timed out before job completed.

Error message

Polling timed out before job completed.

What it means

TimeoutError raised by the dataset tool's while/else construct: the polling loop ran for the full `timeout` seconds without the snapshot reaching 'ready' (and without an explicit error status). Note the loop sleeps polling_interval before each check, so elapsed time accumulates in discrete chunks.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/brightdata_tool/brightdata_dataset.py:535

                async with session.get(
                    f"{BRIGHTDATA_API_URL}/datasets/v3/progress/{snapshot_id}",
                    headers=headers,
                ) as status_response:
                    if status_response.status != 200:
                        raise BrightDataDatasetToolException(
                            f"Status check failed: {await status_response.text()}",
                            status_response.status,
                        )
                    status_data = await status_response.json()
                    if status_data.get("status") == "ready":
                        break
                    if status_data.get("status") == "error":
                        raise BrightDataDatasetToolException(
                            f"Job failed: {status_data}", 0
                        )
            else:
                raise TimeoutError("Polling timed out before job completed.")

            async with session.get(
                f"{BRIGHTDATA_API_URL}/datasets/v3/snapshot/{snapshot_id}",
                params={"format": output_format},
                headers=headers,
            ) as snapshot_response:
                if snapshot_response.status != 200:
                    raise BrightDataDatasetToolException(
                        f"Result fetch failed: {await snapshot_response.text()}",
                        snapshot_response.status,
                    )

                return await snapshot_response.text()

    def _run(
        self,
        url: str | None = None,
        dataset_type: str | None = None,

View on GitHub (pinned to 754d7323be)

Solutions

1. Increase the timeout argument (and optionally polling_interval) when requesting large or slow datasets.
2. Check the snapshot in the Bright Data dashboard — if stuck, discard and re-trigger.
3. Catch TimeoutError at the call site (the tool re-raises it after wrapping) and retry or degrade gracefully.
4. Split very large requests into smaller batches so each snapshot finishes within timeout.

Example fix

# before
result = tool.run(url=url, dataset_type='amazon_product')  # default timeout

# after
result = tool.run(url=url, dataset_type='amazon_product', timeout=600, polling_interval=10)

try:
    result = tool.run(url=url, dataset_type=dataset_type, timeout=300)
except TimeoutError:
    # snapshot may still complete later; retry once with a longer budget
    result = tool.run(url=url, dataset_type=dataset_type, timeout=900, polling_interval=15)

When it happens

Trigger: Calling get_dataset_data with default timeout while the dataset legitimately takes longer (large Amazon/map datasets can take minutes); a stuck snapshot that never transitions from 'running'/'pending' to ready or error.

Common situations: Deep-scrape dataset types under load, timeout parameter not scaled with input size, short defaults in library versions vs slower Bright Data processing.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Unable to find the dataset for {dataset_type}. Please make s
• Trigger failed: {await trigger_response.text()}
• Status check failed: {await status_response.text()}
• Job failed: {status_data}
• dataset_type is required either in constructor or method cal

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/2b59be712459ded6. Report an issue: GitHub.