Mintplex-Labs/anything-llm · warning · Error

URL could not be scraped and no content was found.

Error message

URL could not be scraped and no content was found.

What it means

Thrown by the agent's `scrape` tool when `new CollectorApi().getLinkContent(url)` resolves with `{ success: false }`. The collector is the document-gathering subsystem (HTTP fetcher + reader + parser pipeline). success:false means every stage failed to produce anything usable. The tool first emits an introspection line so the LLM sees the page cannot be used, then throws this hard error.

Source

Thrown at server/utils/agents/aibitat/plugins/web-scraping.js:108

           * Scrape a website and summarize the content based on objective if the content is too large.
           * Objective is the original objective & task that user give to the agent, url is the url of the website to be scraped.
           * Here we can leverage the document collector to get raw website text quickly.
           *
           * @param url
           * @returns
           */
          scrape: async function (url) {
            this.super.introspect(
              `${this.caller}: Scraping the content of ${url}`
            );
            const { success, content } =
              await new CollectorApi().getLinkContent(url);

            if (!success) {
              this.super.introspect(
                `${this.caller}: could not scrape ${url}. I can't use this page's content.`
              );
              throw new Error(
                `URL could not be scraped and no content was found.`
              );
            }

            if (!content || content?.length === 0) {
              throw new Error("There was no content to be collected or read.");
            }

            this.reportUrlCitation(url, content);
            const { TokenManager } = require("../../../helpers/tiktoken");
            const tokenEstimate = new TokenManager(
              this.super.model
            ).countFromString(content);
            if (
              tokenEstimate <
              Provider.contextLimit(this.super.provider, this.super.model)
            ) {
              this.super.introspect(

View on GitHub (pinned to 526360e320)

Solutions

1. Confirm the collector service/container is running and reachable from the server process.
2. Open the URL in a browser to confirm it is publicly readable and not paywalled or login-gated.
3. Try the site's canonical, printer-friendly, or AMP version of the same page.
4. If the collector supports it, enable JavaScript rendering for SPAs.
5. Check the collector's own logs for the underlying fetch failure (timeout, 403, TLS) for this specific URL.

// Cheap preflight: HEAD the URL to catch obvious 4xx/5xx before invoking the collector.
async function isUrlLikelyFetchable(url) {
  try {
    const res = await fetch(url, { method: 'HEAD', redirect: 'follow' });
    return res.ok;
  } catch { return false; }
}

// Narrow the collector response.
function isCollectorSuccess(result) {
  return result != null && typeof result === 'object' && result.success === true && typeof result.content === 'string';
}

// Catch at the tool boundary so the agent can fall back to search snippets.
try {
  return await scrape(url);
} catch (e) {
  this.super.introspect(`${this.caller}: scrape failed (${e.message}); using search snippet instead.`);
  return null; // let the caller choose a different source
}

When it happens

Trigger: Target URL returns 404/403/500; URL is a JavaScript-only SPA the collector cannot render; URL is behind a paywall or login; URL is a binary (image/video) with no extractable text; URL redirects to a disallowed host; the collector microservice itself is down or unreachable.

Common situations: User pastes a paywalled news link; the site blocks the collector's User-Agent; the collector container is not running (so getLinkContent reports failure for every URL); URL is a raw Google Docs or dynamic canvas app; TLS certificate problem on the target host.

Related errors

• There was no content to be collected or read.
• URL could not be scraped and no content was found.
• HTTP ${res.status}: ${res.statusText}
• Failed to fetch document content: ${response.status}
• Audio conversion failed.

AI-assisted analysis of Mintplex-Labs/anything-llm@526360e320 (2026-08-13). Data as JSON: /api/errors/8c3630d036042cb2. Report an issue: GitHub.