unclecode/crawl4ai · error · HTTPException

str(e)

Error message

str(e)

What it means

POST /llm/job (llm_job_enqueue in deploy/docker/job.py) validates payload.webhook_config.webhook_url via utils.validate_webhook_url; a ValueError becomes HTTPException 400 with the validator's message in detail. This fails fast before the 202 job is accepted, so bad webhook URLs never produce jobs whose completion callback silently fails.

Source

Thrown at deploy/docker/job.py:81

    crawler_config: Dict = {}
    webhook_config: Optional[WebhookConfig] = None


# ---------- LL​M job ---------------------------------------------------------
@router.post("/llm/job", status_code=202)
async def llm_job_enqueue(
        payload: LlmJobPayload,
        background_tasks: BackgroundTasks,
        request: Request,
        _td: Optional[Dict] = Depends(_principal_dep),
):
    webhook_config = None
    if payload.webhook_config:
        from utils import validate_webhook_url
        try:
            validate_webhook_url(str(payload.webhook_config.webhook_url))
        except ValueError as e:
            raise HTTPException(status_code=400, detail=str(e))
        webhook_config = payload.webhook_config.model_dump(mode='json')

    return await handle_llm_request(
        _redis,
        background_tasks,
        request,
        str(payload.url),
        query=payload.q,
        schema=payload.schema,
        cache=payload.cache,
        config=_config,
        provider=payload.provider,
        webhook_config=webhook_config,
        temperature=payload.temperature,
        requester=_owner_of(_td),
        is_admin=_is_admin(_td),
    )

View on GitHub (pinned to 7e80152142)

Solutions

1. Read detail in the 400 response — validate_webhook_url's message states the specific rule violated.
2. Supply a fully-qualified https:// URL on a public host for webhook_url.
3. If you don't need callbacks, omit webhook_config entirely.
4. Re-submit the job after fixing the URL; no partial state is created since validation precedes enqueue.

Example fix

# before
{"webhook_config": {"webhook_url": "hooks.mycompany.internal/jobs"}}

# after
{"webhook_config": {"webhook_url": "https://hooks.mycompany.com/jobs"}}

from urllib.parse import urlparse

def webhook_ok(url: str) -> bool:
    p = urlparse(url)
    return p.scheme in ("http", "https") and bool(p.netloc) and "localhost" not in p.netloc and not p.hostname.startswith("127.")

def is_submittable_webhook(url) -> bool:
    try:
        p = urlparse(url)
        return p.scheme == "https" and "." in (p.hostname or "")
    except Exception:
        return False

resp = client.post("/llm/job", json=payload)
if resp.status_code == 400:
    fix_webhook_from_detail(resp.json()["detail"])  # message states the violated rule
    resp = client.post("/llm/job", json=payload)

When it happens

Trigger: Submitting an LLM extraction job with webhook_config.webhook_url that is not a valid http(s) URL — e.g. missing scheme, ftp:// scheme, a hostname the validator considers internal/unresolvable, or a malformed URL string.

Common situations: Webhook set to an internal/staging host (blocked to prevent SSRF); typo like 'hooks.example' without scheme; trailing whitespace or quotes around the URL pasted from a secrets manager; using a localhost webhook in production.

Related errors

• Invalid status: {status}. Must be one of: all, active, compl
• Container not found: ${config.container_selector}
• str(e)
• error_msg
• Invalid limit: {limit}. Must be between 1 and 1000

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/1aa187be577c8ad8. Report an issue: GitHub.