Mintplex-Labs/anything-llm · error · Error

URL could not be scraped and no content was found.

Error message

URL could not be scraped and no content was found.

What it means

Thrown when CollectorApi.getLinkContent() resolves with success:false, meaning the collector service attempted to fetch the URL but could not retrieve usable content. The collector is the AnythingLLM document-fetching backend; success:false indicates a retrieval-level failure distinct from an empty-but-successful page.

Source

Thrown at server/utils/agentFlows/executors/web-scraping.js:34

    `\x1b[43m[AgentFlowToolExecutor]\x1b[0m - executing Web Scraping block`
  );

  if (!url) {
    throw new Error("URL is required for web scraping");
  }

  const captureMode = captureAs === "querySelector" ? "html" : captureAs;
  introspect(`Scraping the content of ${url} as ${captureAs}`);
  const { success, content } = await new CollectorApi()
    .getLinkContent(url, captureMode)
    .then((res) => {
      if (captureAs !== "querySelector") return res;
      return parseHTMLwithSelector(res.content, config.querySelector, context);
    });

  if (!success) {
    introspect(`Could not scrape ${url}. Cannot use this page's content.`);
    throw new Error("URL could not be scraped and no content was found.");
  }

  introspect(`Successfully scraped content from ${url}`);
  if (!content || content?.length === 0) {
    introspect("There was no content to be collected or read.");
    throw new Error("There was no content to be collected or read.");
  }

  if (!enableSummarization) {
    logger(`Returning raw content as summarization is disabled`);
    return content;
  }

  const tokenCount = new TokenManager(
    aibitat.defaultProvider.model
  ).countFromString(content);
  const contextLimit = Provider.contextLimit(
    aibitat.defaultProvider.provider,

View on GitHub (pinned to 526360e320)

Solutions

1. Open the URL in a browser to confirm it loads and has static HTML content.
2. Verify the collector service is running and reachable from the server.
3. Try a different, simpler URL to isolate whether the issue is site-specific.
4. If using querySelector, confirm the selector matches elements in the page's HTML.

// pre-flight: basic URL reachability is not guaranteed, but validate shape
function validateScrapeUrl(url) {
  try { const u = new URL(url); if (!u.protocol.startsWith("http")) throw 0; }
  catch { throw new Error(`Invalid scrape url: ${url}`); }
}

try {
  result = await executeWebScraping(config, context);
} catch (error) {
  if (error.message.includes("could not be scraped")) {
    // fall back to an alternative source or return a graceful failure
  } else throw error;
}

When it happens

Trigger: The URL is unreachable, returns an HTTP error, is blocked by anti-bot protection, requires JavaScript rendering that the collector does not perform, the collector service is down, or the URL redirects to an unsupported resource type. For querySelector mode, parseHTMLwithSelector returning { success:false } (no elements matched) also surfaces here.

Common situations: Target site blocks the collector's user-agent; the page is a JS-only SPA with no server-rendered HTML; the collector process is not running (Docker setup without the collector); a CSS querySelector that matches no elements.

Related errors

• ${res.error || "Failed to save flow"}
• ${res.error || "Failed to get flow"}
• ${res.error || "Failed to delete flow"}
• ${res.error || "Failed to toggle flow"}
• URL is required for web scraping

AI-assisted analysis of Mintplex-Labs/anything-llm@526360e320 (2026-08-13). Data as JSON: /api/errors/83d6eb7b508482d1. Report an issue: GitHub.