{"record":{"id":"d3354834161ef8eb","repo":"epi052/feroxbuster","slug":"could-not-parse","errorCode":null,"errorMessage":"Could not parse {}: {}","messagePattern":"Could not parse (.+?): (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/extractor/container.rs","lineNumber":491,"sourceCode":"        paths\n    }\n\n    /// simple helper to stay DRY, tries to join a url + fragment and add it to the `links` HashSet\n    pub(super) fn add_link_to_set_of_links(\n        &self,\n        link: &str,\n        links: &mut HashSet<String>,\n    ) -> Result<()> {\n        log::trace!(\"enter: add_link_to_set_of_links({link}, {links:?})\");\n\n        let old_url = match self.target {\n            ExtractionTarget::ResponseBody | ExtractionTarget::DirectoryListing => {\n                self.response.unwrap().url().clone()\n            }\n            ExtractionTarget::RobotsTxt => match parse_url_with_raw_path(&self.url) {\n                Ok(u) => u,\n                Err(e) => {\n                    bail!(\"Could not parse {}: {}\", self.url, e);\n                }\n            },\n        };\n\n        let new_url = old_url\n            .join(link)\n            .with_context(|| format!(\"Could not join {old_url} with {link}\"))?;\n\n        if !new_url.is_in_scope(&self.handles.config.scope) {\n            // URL is not in scope based on domain/scope configuration\n            log::debug!(\"Skipping {new_url} because it's not in scope\");\n            log::trace!(\"exit: add_link_to_set_of_links\");\n            return Ok(());\n        }\n\n        links.insert(new_url.to_string());\n\n        log::trace!(\"exit: add_link_to_set_of_links\");","sourceCodeStart":473,"sourceCodeEnd":509,"githubUrl":"https://github.com/epi052/feroxbuster/blob/1f595dab5c76858d5a14fbc47dabf2563d729c62/src/extractor/container.rs#L473-L509","documentation":"This error is thrown by the extractor's add_link_to_set_of_links when it tries to derive the base URL for the ExtractionTarget::RobotsTxt variant by calling parse_url_with_raw_path on the stored URL, and that parsing fails. The library cannot establish a joinable base URL for links extracted from a robots.txt response, so it aborts extraction of that link with this bail! message. It is a URL-format failure inside the link-extraction pipeline.","triggerScenarios":"Scanning with robots.txt extraction enabled and the target's stored URL is malformed or contains a raw path component that parse_url_with_raw_path cannot parse (e.g. an oddly formed base URL passed to the scanner, or a URL with invalid characters/encoding) so base-url resolution fails before join(link).","commonSituations":"Feeding the scanner a hand-typed or proxied base URL with stray characters (spaces, duplicate schemes like http://http://, or bare paths), running behind a redirect/proxy that rewrites URLs into non-standard forms, or a target host whose robots.txt flow receives a URL that survived as an unparsable string.","solutions":["Verify the base URL passed via the target/URL argument is a well-formed absolute URL (scheme + host, no stray spaces or duplicate schemes)","Use the plain host form (e.g. https://example.com) instead of a path-bearing or rewritten URL when robots.txt extraction is enabled","Reproduce parse_url_with_raw_path against the failing URL locally to see the underlying parse error included in the message","Update to the latest version in case the URL parser has fixes for edge-case encodings"],"exampleFix":"// before\nferox --url \"example.com\"  // no scheme, robots.txt parsing fails\n// after\nferox --url \"https://example.com\"","handlingStrategy":"validation","validationCode":"if let Err(e) = url::Url::parse(target_url) { eprintln!(\"fix base URL: {e}\"); return; }","typeGuard":"fn is_parsable_url(s: &str) -> bool { url::Url::parse(s).is_ok() }","tryCatchPattern":null,"preventionTips":["Always pass absolute, well-formed URLs with scheme and host","Avoid hand-typed or proxy-rewritten URLs as scan targets","Test the target URL with a quick parse/curl before scanning"],"tags":["url-parsing","extraction","robots-txt","rust"],"backgroundTag":"invalid-url-format","analyzedSha":"1f595dab5c76858d5a14fbc47dabf2563d729c62","analyzedAt":"2026-09-13T19:33:06.208Z","contentChangedAt":"2026-09-13T19:33:06.208Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}