crewAIInc/crewAI · error · BrightDataDatasetToolException

Job failed: {status_data}

Error message

Job failed: {status_data}

What it means

A BrightDataDatasetToolException raised when the progress endpoint reports status == 'error' for the snapshot. The submission and polling both worked, but Bright Data itself failed to complete the scrape job; status_data carries Bright Data's error details and the tool passes status code 0.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/brightdata_tool/brightdata_dataset.py:531

            elapsed = 0
            while elapsed < timeout:
                await asyncio.sleep(polling_interval)
                elapsed += polling_interval

                async with session.get(
                    f"{BRIGHTDATA_API_URL}/datasets/v3/progress/{snapshot_id}",
                    headers=headers,
                ) as status_response:
                    if status_response.status != 200:
                        raise BrightDataDatasetToolException(
                            f"Status check failed: {await status_response.text()}",
                            status_response.status,
                        )
                    status_data = await status_response.json()
                    if status_data.get("status") == "ready":
                        break
                    if status_data.get("status") == "error":
                        raise BrightDataDatasetToolException(
                            f"Job failed: {status_data}", 0
                        )
            else:
                raise TimeoutError("Polling timed out before job completed.")

            async with session.get(
                f"{BRIGHTDATA_API_URL}/datasets/v3/snapshot/{snapshot_id}",
                params={"format": output_format},
                headers=headers,
            ) as snapshot_response:
                if snapshot_response.status != 200:
                    raise BrightDataDatasetToolException(
                        f"Result fetch failed: {await snapshot_response.text()}",
                        snapshot_response.status,
                    )

                return await snapshot_response.text()

View on GitHub (pinned to 754d7323be)

Solutions

1. Inspect status_data in the exception message for Bright Data's failure reason.
2. Verify the target URL loads in a browser and is reachable from the dataset's configured zone.
3. Re-trigger with corrected input; if the failure repeats, test the same payload in the Bright Data dashboard.
4. For flaky targets, retry the full run once — intermittent blocks sometimes clear on retry.

import urllib.request

def url_reachable(url: str) -> bool:
    try:
        req = urllib.request.Request(url, method="HEAD", headers={"User-Agent": "Mozilla/5.0"})
        with urllib.request.urlopen(req, timeout=10) as r:
            return r.status < 400
    except Exception:
        return False

if not url_reachable(url):
    raise ValueError(f"Target unreachable, Bright Data job would fail: {url}")

try:
    result = tool.run(url=url, dataset_type=dataset_type)
except BrightDataDatasetToolException as e:
    if "Job failed" in str(e):
        # fall back to the unlocker tool for a raw page fetch
        result = unlocker_tool.run(url=url, data_format='markdown')
    else:
        raise

When it happens

Trigger: Bright Data could not scrape the target URL (blocked, removed page, geo-restricted), invalid input for the dataset type accepted at trigger time but rejected at execution, or upstream dataset runner failures.

Common situations: Scraping URLs that block datacenter proxies, product pages that 404 between trigger and execution, malformed additional_params that only surface at runtime.

Related errors

• Unable to find the dataset for {dataset_type}. Please make s
• Trigger failed: {await trigger_response.text()}
• Polling timed out before job completed.
• dataset_type is required either in constructor or method cal
• Unsupported output format: {output_format}. Must be one of {

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/4890e0e1d06a136b. Report an issue: GitHub.