{"record":{"id":"905314f08295dcf8","repo":"unclecode/crawl4ai","slug":"invalid-url-format-must-start-with-http-https","errorCode":null,"errorMessage":"Invalid URL format. Must start with http://, https://, or for raw HTML (raw:, raw://)","messagePattern":"Invalid URL format\\. Must start with http://, https://, or for raw HTML \\(raw:, raw://\\)","errorType":"http","errorClass":"HTTPException","httpStatus":400,"severity":"error","filePath":"deploy/docker/server.py","lineNumber":583,"sourceCode":"@mcp_tool(\"md\")\nasync def get_markdown(\n request: Request,\n body: MarkdownRequest,\n _td: Dict = Depends(token_dep),\n):\n \"\"\"\n Convert a web page into Markdown format.\n\n Supports multiple extraction modes:\n - fit (default): Readability-based extraction for clean content\n - raw: Direct DOM to Markdown conversion\n - bm25: BM25 relevance ranking with optional query\n - llm: LLM-based summarization with optional query\n\n Use this tool when you need clean, readable text from web pages.\n \"\"\"\n if not body.url.startswith((\"http://\", \"https://\")) and not body.url.startswith((\"raw:\", \"raw://\")):\n raise HTTPException(\n 400, \"Invalid URL format. Must start with http://, https://, or for raw HTML (raw:, raw://)\")\n # base_url is intentionally not accepted from the request (key-exfil vector);\n # the LLM endpoint is server-derived from the provider name only.\n markdown = await handle_markdown_request(\n body.url, body.f, body.q, body.c, config, body.provider,\n body.temperature\n )\n return JSONResponse({\n \"url\": body.url,\n \"filter\": body.f,\n \"query\": body.q,\n \"cache\": body.c,\n \"markdown\": markdown,\n \"success\": True\n })\n\n\n@app.post(\"/html\")","sourceCodeStart":565,"sourceCodeEnd":601,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/deploy/docker/server.py#L565-L601","documentation":"A 400 from POST /md: the url field must start with http://, https://, or a raw-HTML prefix (raw: or raw://). The endpoint converts pages to Markdown and only accepts remote URLs or inline raw HTML — bare hostnames, file://, data:, etc. are rejected before any crawl.","triggerScenarios":"POST /md with body.url = 'example.com/page' (no scheme), 'file:///tmp/x.html', 'ftp://...', or any string lacking the four accepted prefixes. The check is a plain startswith, so even whitespace or a leading '/' breaks it.","commonSituations":"Users pasting URLs without the scheme; passing a local file path expecting the server to read it; forgetting the 'raw:' prefix when submitting inline HTML for conversion.","solutions":["Prefix the URL with https:// (e.g. 'https://example.com/page').","For inline HTML, pass the content as 'raw:...' or 'raw://...'.","Normalize/strip whitespace on the client before sending.","Use a dedicated URL parser client-side (urllib.parse.urlparse) to confirm scheme is http/https first."],"exampleFix":"# before\nbody = {'url': 'example.com/article'}\n# after\nfrom urllib.parse import urlparse\nu = 'example.com/article'\nbody = {'url': u if urlparse(u).scheme in ('http','https') else 'https://' + u}","handlingStrategy":"validation","validationCode":"from urllib.parse import urlparse\n\ndef valid_md_url(url: str) -> bool:\n u = url.strip()\n return u.startswith(('http://', 'https://', 'raw:', 'raw://'))\n\ndef normalize(url: str) -> str:\n u = url.strip()\n if not u.startswith(('http://', 'https://', 'raw:', 'raw://')):\n u = 'https://' + u\n return u","typeGuard":"def is_md_url(url: str) -> bool:\n return isinstance(url, str) and url.strip().startswith(('http://', 'https://', 'raw:', 'raw://'))","tryCatchPattern":null,"preventionTips":["Strip whitespace and always normalize to https:// before sending.","Use 'raw:...' for inline HTML instead of file paths.","Centralize URL normalization in one client helper so every endpoint gets the same treatment."],"tags":["url-validation","http-400","markdown"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}