{"record":{"id":"7419a6ef148af826","repo":"epi052/feroxbuster","slug":"prevented-request-to-due-to","errorCode":null,"errorMessage":"prevented request to {} due to {:?} || {:?}","messagePattern":"prevented request to (.+?) due to (.+?) \\|\\| (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/extractor/container.rs","lineNumber":51,"sourceCode":"\n    let ferox_url = FeroxUrl::from_string(url, handles.clone());\n\n    // create a url based on the given command line options\n    let new_url = ferox_url.format(\"\", None)?;\n\n    let scanned_urls = handles.ferox_scans()?;\n\n    if scanned_urls.get_scan_by_url(new_url.as_ref()).is_some() {\n        //we've seen the url before and don't need to scan again\n        log::trace!(\"exit: request_link -> None\");\n        bail!(\"previously seen url\");\n    }\n\n    if (!handles.config.url_denylist.is_empty() || !handles.config.regex_denylist.is_empty())\n        && should_deny_url(&new_url, handles.clone())?\n    {\n        // can't allow a denied url to be requested\n        bail!(\n            \"prevented request to {} due to {:?} || {:?}\",\n            url,\n            handles.config.url_denylist,\n            handles.config.regex_denylist,\n        );\n    }\n\n    // make the request and store the response\n    let new_response = logged_request(&new_url, DEFAULT_METHOD, None, handles.clone()).await?;\n\n    log::trace!(\"exit: request_link -> {new_response:?}\");\n\n    Ok(new_response)\n}\n\n/// Whether an active scan is recursive or not\n#[derive(Debug, Copy, Clone)]\nenum RecursionStatus {","sourceCodeStart":33,"sourceCodeEnd":69,"githubUrl":"https://github.com/epi052/feroxbuster/blob/1f595dab5c76858d5a14fbc47dabf2563d729c62/src/extractor/container.rs#L33-L69","documentation":"request_link checks the target URL against the configured denylists before making a request; if should_deny_url matches (by URL pattern or regex), the request is refused with this error naming the URL and both denylist contents. It prevents the link extractor from fetching URLs the user excluded.","triggerScenarios":"While crawling (request_links), the extractor discovers a link that matches --url-denylist or --regex-denylist (or config-file equivalents) and attempts to request it, triggering the bail.","commonSituations":"Users setting broad denylist patterns that also match links extracted from allowed pages (e.g. denying '/logout' that appears as a link everywhere); overly general regex like '.*admin.*' catching crawled pages; recursive scans following links into denied paths.","solutions":["Review and narrow the url_denylist/regex_denylist values shown in the error message so intended links are not matched","Anchor regex denylist patterns (e.g. '^https://host/admin/') instead of broad substrings","If the URL should be scanned, remove it from the denylist or use a more specific entry","Accept the error as expected behavior if the denial is intentional; skip/ignore those links"],"exampleFix":"// before\nregex_denylist: [\".*admin.*\"]  // matches crawled links like /admin-blog\n// after\nregex_denylist: [\"^https://target/admin/\"]","handlingStrategy":"try-catch","validationCode":"use ferox_scanner::url::should_deny_url;\nif should_deny_url(&link_url, handles.clone())? {\n    log::info!(\"skipping denied link: {link_url}\");\n    return Ok(()); // skip before calling request_link\n}","typeGuard":null,"tryCatchPattern":"match extractor.request_link(&link_url).await {\n    Ok(_) => {}\n    Err(e) if e.to_string().starts_with(\"prevented request to\") => {\n        log::debug!(\"link denied by denylist, skipping: {link_url}\");\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Keep denylist patterns anchored and specific; test them against expected crawled links","Check the url/regex denylist values printed in the error when a link is unexpectedly skipped","Treat this bail as informational during recursive scans when denial is intentional","Review config-file denylist entries after schema/version changes"],"tags":["denylist","crawler","request-filtering"],"backgroundTag":"permission-denied","analyzedSha":"1f595dab5c76858d5a14fbc47dabf2563d729c62","analyzedAt":"2026-09-13T19:33:06.208Z","contentChangedAt":"2026-09-13T19:33:06.208Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}