langgenius/dify · error · WebsiteCrawlError

crawl_failed

crawl_failed

Error message

{message}

What it means

Raised as WebsiteCrawlError (error_code crawl_failed, HTTP 500) at website.py:48 when WebsiteCrawlApiRequest.from_args throws a ValueError during request construction in POST /website/crawl. The controller wraps the validation ValueError into WebsiteCrawlError, whose description template is '{message}'. Although the root cause is bad input, the response status is 500 because WebsiteCrawlError.code is 500.

Source

Thrown at api/controllers/console/datasets/website.py:48

@console_ns.route("/website/crawl")
class WebsiteCrawlApi(Resource):
    @console_ns.doc("crawl_website")
    @console_ns.doc(description="Crawl website content")
    @console_ns.expect(console_ns.models[WebsiteCrawlPayload.__name__])
    @console_ns.response(200, "Website crawl initiated successfully", console_ns.models[WebsiteCrawlResponse.__name__])
    @console_ns.response(400, "Invalid crawl parameters")
    @setup_required
    @login_required
    @account_initialization_required
    @model_validate(WebsiteCrawlPayload)
    def post(self, req_data: WebsiteCrawlPayload):
        # Create typed request and validate
        try:
            api_request = WebsiteCrawlApiRequest.from_args(req_data.model_dump())
        except ValueError as e:
            raise WebsiteCrawlError(str(e))

        # Crawl URL using typed request
        try:
            result = WebsiteService.crawl_url(api_request)
        except Exception as e:
            raise WebsiteCrawlError(str(e))
        return result, 200


@console_ns.route("/website/crawl/status/<string:job_id>")
class WebsiteCrawlStatusApi(Resource):
    @console_ns.doc("get_crawl_status")
    @console_ns.doc(description="Get website crawl status")
    @console_ns.doc(params={"job_id": "Crawl job ID", "provider": "Crawl provider (firecrawl/watercrawl/jinareader)"})
    @console_ns.doc(params=query_params_from_model(WebsiteCrawlStatusQuery))
    @console_ns.response(200, "Crawl status retrieved successfully", console_ns.models[WebsiteCrawlResponse.__name__])
    @console_ns.response(404, "Crawl job not found")
    @console_ns.response(400, "Invalid provider")

View on GitHub (pinned to ef8544b173)

Solutions

1. Ensure provider is exactly one of 'firecrawl', 'watercrawl', 'jinareader'.
2. Send a fully-qualified URL (https://...) in the `url` field.
3. Validate the payload against WebsiteCrawlPayload + the from_args rules on the client before posting.
4. Configure the chosen provider's credentials in Settings; a missing/invalid config can also surface here.

Example fix

// before
POST /website/crawl { provider: 'scraper', url: 'example.com' }   // -> 500 crawl_failed
// after
POST /website/crawl { provider: 'firecrawl', url: 'https://example.com', options: {} }

const ALLOWED_PROVIDERS = ['firecrawl', 'watercrawl', 'jinareader'];
function crawlPayload(provider, url, options = {}) {
  if (!ALLOWED_PROVIDERS.includes(provider)) throw new Error(`unsupported provider: ${provider}`);
  try { new URL(url); } catch { throw new Error(`invalid url: ${url}`); }
  return { provider, url, options };
}

function isCrawlPayload(p) {
  return !!p && ['firecrawl','watercrawl','jinareader'].includes(p.provider) && typeof p.url === 'string' && URL.canParse(p.url);
}

try { await fetch('/website/crawl', { method:'POST', headers:{'Content-Type':'application/json'}, body: JSON.stringify(crawlPayload(provider, url, options)) }); } catch (e) { if (e.code === 'crawl_failed') surfaceMessage(e.message); else throw e; }

When it happens

Trigger: POST /website/crawl with a payload whose `provider` is not one of firecrawl/watercrawl/jinareader, whose `url` is empty/malformed, or whose `options` violate WebsiteCrawlApiRequest.from_args validation. from_args raises ValueError, rethrown as WebsiteCrawlError.

Common situations: Provider misspelled or unsupported; URL missing scheme/host; options shape changed across versions; client sending provider values the backend does not yet know.

Related errors

• Invalid filters
• streaming response body missing
• usage_missing_arg
• usage_missing_arg
• usage_invalid_flag

AI-assisted analysis of langgenius/dify@ef8544b173 (2026-08-12). Data as JSON: /api/errors/2caa18db627fbc55. Report an issue: GitHub.