epi052/feroxbuster · error

Could not parse

Error message

Could not parse {}: {}

What it means

This error is thrown by the extractor's add_link_to_set_of_links when it tries to derive the base URL for the ExtractionTarget::RobotsTxt variant by calling parse_url_with_raw_path on the stored URL, and that parsing fails. The library cannot establish a joinable base URL for links extracted from a robots.txt response, so it aborts extraction of that link with this bail! message. It is a URL-format failure inside the link-extraction pipeline.

Solutions

1. Verify the base URL passed via the target/URL argument is a well-formed absolute URL (scheme + host, no stray spaces or duplicate schemes)
2. Use the plain host form (e.g. https://example.com) instead of a path-bearing or rewritten URL when robots.txt extraction is enabled
3. Reproduce parse_url_with_raw_path against the failing URL locally to see the underlying parse error included in the message
4. Update to the latest version in case the URL parser has fixes for edge-case encodings

Example fix

// before
ferox --url "example.com"  // no scheme, robots.txt parsing fails
// after
ferox --url "https://example.com"

if let Err(e) = url::Url::parse(target_url) { eprintln!("fix base URL: {e}"); return; }

fn is_parsable_url(s: &str) -> bool { url::Url::parse(s).is_ok() }

When it happens

Trigger: Scanning with robots.txt extraction enabled and the target's stored URL is malformed or contains a raw path component that parse_url_with_raw_path cannot parse (e.g. an oddly formed base URL passed to the scanner, or a URL with invalid characters/encoding) so base-url resolution fails before join(link).

Common situations: Feeding the scanner a hand-typed or proxied base URL with stray characters (spaces, duplicate schemes like http://http://, or bare paths), running behind a redirect/proxy that rewrites URLs into non-standard forms, or a target host whose robots.txt flow receives a URL that survived as an unparsable string.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

• Empty query string provided
• Empty key in query string
• Empty header provided
• Empty header name provided
• Empty --request-file file provided

AI-assisted analysis of epi052/feroxbuster@1f595dab5c (2026-09-13). Data as JSON: /api/errors/d3354834161ef8eb. Report an issue: GitHub.

Appendix: source

Thrown at src/extractor/container.rs:491

        paths
    }

    /// simple helper to stay DRY, tries to join a url + fragment and add it to the `links` HashSet
    pub(super) fn add_link_to_set_of_links(
        &self,
        link: &str,
        links: &mut HashSet<String>,
    ) -> Result<()> {
        log::trace!("enter: add_link_to_set_of_links({link}, {links:?})");

        let old_url = match self.target {
            ExtractionTarget::ResponseBody | ExtractionTarget::DirectoryListing => {
                self.response.unwrap().url().clone()
            }
            ExtractionTarget::RobotsTxt => match parse_url_with_raw_path(&self.url) {
                Ok(u) => u,
                Err(e) => {
                    bail!("Could not parse {}: {}", self.url, e);
                }
            },
        };

        let new_url = old_url
            .join(link)
            .with_context(|| format!("Could not join {old_url} with {link}"))?;

        if !new_url.is_in_scope(&self.handles.config.scope) {
            // URL is not in scope based on domain/scope configuration
            log::debug!("Skipping {new_url} because it's not in scope");
            log::trace!("exit: add_link_to_set_of_links");
            return Ok(());
        }

        links.insert(new_url.to_string());

        log::trace!("exit: add_link_to_set_of_links");

View on GitHub (pinned to 1f595dab5c)