{"record":{"id":"2caa18db627fbc55","repo":"langgenius/dify","slug":"crawl-failed","errorCode":"crawl_failed","errorMessage":"{message}","messagePattern":"\\{message\\}","errorType":"error_code","errorClass":"WebsiteCrawlError","httpStatus":500,"severity":"error","filePath":"api/controllers/console/datasets/website.py","lineNumber":48,"sourceCode":"\n\n@console_ns.route(\"/website/crawl\")\nclass WebsiteCrawlApi(Resource):\n    @console_ns.doc(\"crawl_website\")\n    @console_ns.doc(description=\"Crawl website content\")\n    @console_ns.expect(console_ns.models[WebsiteCrawlPayload.__name__])\n    @console_ns.response(200, \"Website crawl initiated successfully\", console_ns.models[WebsiteCrawlResponse.__name__])\n    @console_ns.response(400, \"Invalid crawl parameters\")\n    @setup_required\n    @login_required\n    @account_initialization_required\n    @model_validate(WebsiteCrawlPayload)\n    def post(self, req_data: WebsiteCrawlPayload):\n        # Create typed request and validate\n        try:\n            api_request = WebsiteCrawlApiRequest.from_args(req_data.model_dump())\n        except ValueError as e:\n            raise WebsiteCrawlError(str(e))\n\n        # Crawl URL using typed request\n        try:\n            result = WebsiteService.crawl_url(api_request)\n        except Exception as e:\n            raise WebsiteCrawlError(str(e))\n        return result, 200\n\n\n@console_ns.route(\"/website/crawl/status/<string:job_id>\")\nclass WebsiteCrawlStatusApi(Resource):\n    @console_ns.doc(\"get_crawl_status\")\n    @console_ns.doc(description=\"Get website crawl status\")\n    @console_ns.doc(params={\"job_id\": \"Crawl job ID\", \"provider\": \"Crawl provider (firecrawl/watercrawl/jinareader)\"})\n    @console_ns.doc(params=query_params_from_model(WebsiteCrawlStatusQuery))\n    @console_ns.response(200, \"Crawl status retrieved successfully\", console_ns.models[WebsiteCrawlResponse.__name__])\n    @console_ns.response(404, \"Crawl job not found\")\n    @console_ns.response(400, \"Invalid provider\")","sourceCodeStart":30,"sourceCodeEnd":66,"githubUrl":"https://github.com/langgenius/dify/blob/ef8544b173fd6cd7a8e71df2cab576e52bebbfbc/api/controllers/console/datasets/website.py#L30-L66","documentation":"Raised as WebsiteCrawlError (error_code crawl_failed, HTTP 500) at website.py:48 when WebsiteCrawlApiRequest.from_args throws a ValueError during request construction in POST /website/crawl. The controller wraps the validation ValueError into WebsiteCrawlError, whose description template is '{message}'. Although the root cause is bad input, the response status is 500 because WebsiteCrawlError.code is 500.","triggerScenarios":"POST /website/crawl with a payload whose `provider` is not one of firecrawl/watercrawl/jinareader, whose `url` is empty/malformed, or whose `options` violate WebsiteCrawlApiRequest.from_args validation. from_args raises ValueError, rethrown as WebsiteCrawlError.","commonSituations":"Provider misspelled or unsupported; URL missing scheme/host; options shape changed across versions; client sending provider values the backend does not yet know.","solutions":["Ensure provider is exactly one of 'firecrawl', 'watercrawl', 'jinareader'.","Send a fully-qualified URL (https://...) in the `url` field.","Validate the payload against WebsiteCrawlPayload + the from_args rules on the client before posting.","Configure the chosen provider's credentials in Settings; a missing/invalid config can also surface here."],"exampleFix":"// before\nPOST /website/crawl { provider: 'scraper', url: 'example.com' }   // -> 500 crawl_failed\n// after\nPOST /website/crawl { provider: 'firecrawl', url: 'https://example.com', options: {} }","handlingStrategy":"validation","validationCode":"const ALLOWED_PROVIDERS = ['firecrawl', 'watercrawl', 'jinareader'];\nfunction crawlPayload(provider, url, options = {}) {\n  if (!ALLOWED_PROVIDERS.includes(provider)) throw new Error(`unsupported provider: ${provider}`);\n  try { new URL(url); } catch { throw new Error(`invalid url: ${url}`); }\n  return { provider, url, options };\n}","typeGuard":"function isCrawlPayload(p) {\n  return !!p && ['firecrawl','watercrawl','jinareader'].includes(p.provider) && typeof p.url === 'string' && URL.canParse(p.url);\n}","tryCatchPattern":"try { await fetch('/website/crawl', { method:'POST', headers:{'Content-Type':'application/json'}, body: JSON.stringify(crawlPayload(provider, url, options)) }); } catch (e) { if (e.code === 'crawl_failed') surfaceMessage(e.message); else throw e; }","preventionTips":["Restrict provider to the three allowed values on the client.","Always send a fully-qualified URL.","Validate before posting to avoid the 500."],"tags":["website","crawl","validation","external-service","http-500"],"backgroundTag":null,"analyzedSha":"ef8544b173fd6cd7a8e71df2cab576e52bebbfbc","analyzedAt":"2026-08-12T05:15:17.394Z","schemaVersion":2},"datasetVersion":"2026-08-12T13:17:24.610Z"}