crewAIInc/crewAI · error · BrightDataDatasetToolException

Status check failed: {await status_response.text()}

Error message

Status check failed: {await status_response.text()}

What it means

A BrightDataDatasetToolException raised during the polling loop when GET /datasets/v3/progress/{snapshot_id} returns a non-200 status. The trigger succeeded and a snapshot id exists, but checking job progress failed — typically auth revocation mid-job, a snapshot that expired on Bright Data's side, or a transient server error.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/brightdata_tool/brightdata_dataset.py:523

                if trigger_response.status != 200:
                    raise BrightDataDatasetToolException(
                        f"Trigger failed: {await trigger_response.text()}",
                        trigger_response.status,
                    )
                trigger_data = await trigger_response.json()
                snapshot_id = trigger_data.get("snapshot_id")

            elapsed = 0
            while elapsed < timeout:
                await asyncio.sleep(polling_interval)
                elapsed += polling_interval

                async with session.get(
                    f"{BRIGHTDATA_API_URL}/datasets/v3/progress/{snapshot_id}",
                    headers=headers,
                ) as status_response:
                    if status_response.status != 200:
                        raise BrightDataDatasetToolException(
                            f"Status check failed: {await status_response.text()}",
                            status_response.status,
                        )
                    status_data = await status_response.json()
                    if status_data.get("status") == "ready":
                        break
                    if status_data.get("status") == "error":
                        raise BrightDataDatasetToolException(
                            f"Job failed: {status_data}", 0
                        )
            else:
                raise TimeoutError("Polling timed out before job completed.")

            async with session.get(
                f"{BRIGHTDATA_API_URL}/datasets/v3/snapshot/{snapshot_id}",
                params={"format": output_format},
                headers=headers,
            ) as snapshot_response:

View on GitHub (pinned to 754d7323be)

Solutions

1. Check the exception's status_code: 401/403 means credentials; 404 means the snapshot id is gone — re-trigger the job.
2. For transient 5xx/429, catch BrightDataDatasetToolException and re-run the whole trigger+poll flow with backoff.
3. Shorten polling_interval / raise timeout only if the dataset legitimately takes long; prefer retrying the full call.

try:
    result = tool.run(url=url, dataset_type=dataset_type)
except BrightDataDatasetToolException as e:
    if e.status_code in (429, 500, 502, 503):
        time.sleep(5)
        result = tool.run(url=url, dataset_type=dataset_type)  # full re-trigger
    elif e.status_code == 404:
        # snapshot gone — must re-trigger, cannot resume
        result = tool.run(url=url, dataset_type=dataset_type)
    else:
        raise

When it happens

Trigger: Snapshot id no longer valid (expired/purged on Bright Data), API key deactivated between trigger and poll, transient 5xx from the progress endpoint during a long-running job.

Common situations: Long-polling jobs that outlive snapshot retention, rotating credentials mid-run, network instability during polling_interval cycles.

Related errors

• Result fetch failed: {await snapshot_response.text()}
• Trigger failed: {await trigger_response.text()}
• Polling timed out before job completed.
• Project name '{name}' would generate invalid Python class na
• Failed to retrieve organization list: {e!s}

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/1b63da90b3078c57. Report an issue: GitHub.