{"record":{"id":"36e709cadff8752f","repo":"Mintplex-Labs/anything-llm","slug":"http-res-status-res-statustext","errorCode":null,"errorMessage":"HTTP ${res.status}: ${res.statusText}","messagePattern":"HTTP (.+?): (.+?)","errorType":"http","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"collector/utils/downloadURIToFile/index.js","lineNumber":46,"sourceCode":" * @returns {Promise<{success: boolean, fileLocation: string|null, reason: string|null}>} - The path to the downloaded file\n */\nasync function downloadURIToFile(url, maxTimeout = 10_000) {\n  if (!url || typeof url !== \"string\" || !validURL(url))\n    return { success: false, reason: \"Not a valid URL.\", fileLocation: null };\n\n  try {\n    const abortController = new AbortController();\n    const timeout = setTimeout(() => {\n      abortController.abort();\n      console.error(\n        `Timeout ${maxTimeout}ms reached while downloading file for URL:`,\n        url.toString()\n      );\n    }, maxTimeout);\n\n    const res = await fetch(url, { signal: abortController.signal })\n      .then((res) => {\n        if (!res.ok) throw new Error(`HTTP ${res.status}: ${res.statusText}`);\n        return res;\n      })\n      .finally(() => clearTimeout(timeout));\n\n    const urlObj = new URL(url);\n    const sluggedPath = slugify(urlObj.pathname, { lower: true });\n    let filename = `${urlObj.hostname}-${sluggedPath}`;\n\n    const existingExt = path.extname(filename).toLowerCase();\n    const { SUPPORTED_FILETYPE_CONVERTERS } = require(\"../constants\");\n\n    // If the filename does not already have a supported file extension,\n    // try to infer one from the response Content-Type header.\n    // This handles URLs like https://arxiv.org/pdf/2307.10265 where the\n    // path has no explicit extension but the server responds with\n    // Content-Type: application/pdf.\n    if (!SUPPORTED_FILETYPE_CONVERTERS.hasOwnProperty(existingExt)) {\n      const { parseContentType } = require(\"../../processLink/helpers\");","sourceCodeStart":28,"sourceCodeEnd":64,"githubUrl":"https://github.com/Mintplex-Labs/anything-llm/blob/526360e320da9d1b36074be5ed64fe76e5bbfbbd/collector/utils/downloadURIToFile/index.js#L28-L64","documentation":"downloadURIToFile fetches a URL and throws if the response is not ok (res.ok === false). The outer try/catch converts the throw into { success: false, reason } — callers see the reason string, not a thrown exception.","triggerScenarios":"Any non-2xx HTTP response: 404 (not found), 403/401 (auth/forbidden), 429 (rate limited), 5xx (server error), or fetch rejecting (DNS, TLS, abort on timeout).","commonSituations":"Link rot (404); paywalled/protected resources (403); rate-limited CDNs (429); transient 5xx; wrong URL pasted; slow server hitting the default 10s timeout.","solutions":["Open the URL in a browser to confirm reachability and the status code.","Add authentication headers if the resource requires them.","Retry on 5xx/429 with backoff; treat 4xx (except 429) as permanent.","Pass a larger maxTimeout for slow sources (default is 10000 ms)."],"exampleFix":"// before\nif (!res.ok) throw new Error(`HTTP ${res.status}: ${res.statusText}`);\n\n// after — include the URL and mark transient errors retryable\nif (!res.ok) {\n  const e = new Error(`HTTP ${res.status} ${res.statusText} for ${url}`);\n  if (res.status >= 500 || res.status === 429) e.retryable = true;\n  throw e;\n}","handlingStrategy":"retry","validationCode":"async function reachable(url, { timeout = 10000 } = {}) {\n  const ac = new AbortController();\n  const t = setTimeout(() => ac.abort(), timeout);\n  try {\n    const r = await fetch(url, { method: \"HEAD\", signal: ac.signal });\n    return r.ok;\n  } catch { return false; }\n  finally { clearTimeout(t); }\n}","typeGuard":null,"tryCatchPattern":"const { success, reason } = await downloadURIToFile(url);\nif (!success) {\n  if (/HTTP 5\\d\\d|429/.test(reason)) { /* retry with backoff */ }\n  else { /* permanent failure — skip */ }\n}","preventionTips":["Validate URLs before queuing downloads.","Retry with backoff on 5xx/429.","Use HEAD requests to pre-check large downloads.","Pass a longer maxTimeout for slow sources."],"tags":["network","http","download","url"],"backgroundTag":null,"analyzedSha":"526360e320da9d1b36074be5ed64fe76e5bbfbbd","analyzedAt":"2026-08-13T01:45:47.170Z","schemaVersion":2},"datasetVersion":"2026-08-13T04:17:16.726Z"}