Mintplex-Labs/anything-llm · error · Error

URL is required for web scraping

Error message

URL is required for web scraping

What it means

Thrown by executeWebScraping when the block's config.url is falsy — missing key, empty string, or a URL built entirely from an unresolvable variable. The check happens before any network activity, right after destructuring { url, captureAs = 'text', enableSummarization = true } from the block config.

Solutions

1. Open the Web Scraping block in the flow builder and set a concrete URL.
2. If the URL uses a {{variable}}, make sure the variable is included in the variables passed to executeFlow / defined by an earlier step.
3. When hand-editing flow JSON, confirm each web_scraping step has "url" set and non-empty.
4. Test with a literal URL first, then switch to the variable once it works.

Example fix

// before: flow step config
{ "type": "web_scraping", "config": { "url": "{{page}}" } }
// variables passed: {} -> Error: URL is required for web scraping

// after
const result = await AgentFlows.executeFlow(uuid, { page: "https://example.com" });

function webScrapingStepConfigOk(step) {
  return Boolean(step.config?.url) && typeof step.config.url === "string";
}

const ready = flow.config.steps
  .filter((s) => s.type === "web_scraping")
  .every(webScrapingStepConfigOk);

/** @param {{config?: {url?: string}}} step */
function hasScrapeUrl(step) {
  return typeof step.config?.url === "string" && step.config.url.trim().length > 0;
}

try {
  await AgentFlows.executeFlow(uuid, variables);
} catch (e) {
  if (/URL is required for web scraping/.test(e.message))
    return respondMissingVariable("web_scraping block needs a url or a filled {{variable}}");
  throw e;
}

When it happens

Trigger: A Web Scraping block whose URL field was left blank in the flow builder; a hand-edited flow JSON missing the url key; or a url referencing a variable (e.g. {{user_url}}) that was never passed in when the flow was executed.

Common situations: Flow template expects variables the caller didn't supply; UI allowed saving the block without a URL (imported JSON); variable name typo means the URL resolves to undefined; API-triggered flow execution (webhook) omitting the variables object.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Device OS and name are required
• Failed to save agent flow.
• Key is required
• Name and config are required
• There was no content to be collected or read.

AI-assisted analysis of Mintplex-Labs/anything-llm@3aec848f28 (2026-08-18). Data as JSON: /api/errors/72a21fbac82250c5. Report an issue: GitHub.

Appendix: source

Thrown at server/utils/agentFlows/executors/web-scraping.js:20

 * Execute a web scraping flow step
 * @param {Object} config Flow step configuration
 * @param {Object} context Execution context with introspect function
 * @returns {Promise<string>} Scraped content
 */
async function executeWebScraping(config, context) {
  const { CollectorApi } = require("../../collectorApi");
  const { TokenManager } = require("../../helpers/tiktoken");
  const Provider = require("../../agents/aibitat/providers/ai-provider");
  const { summarizeContent } = require("../../agents/aibitat/utils/summarize");

  const { url, captureAs = "text", enableSummarization = true } = config;
  const { introspect, logger, aibitat } = context;
  logger(
    `\x1b[43m[AgentFlowToolExecutor]\x1b[0m - executing Web Scraping block`
  );

  if (!url) {
    throw new Error("URL is required for web scraping");
  }

  const captureMode = captureAs === "querySelector" ? "html" : captureAs;
  introspect(`Scraping the content of ${url} as ${captureAs}`);
  const { success, content } = await new CollectorApi()
    .getLinkContent(url, captureMode)
    .then((res) => {
      if (captureAs !== "querySelector") return res;
      return parseHTMLwithSelector(res.content, config.querySelector, context);
    });

  if (!success) {
    introspect(`Could not scrape ${url}. Cannot use this page's content.`);
    throw new Error("URL could not be scraped and no content was found.");
  }

  introspect(`Successfully scraped content from ${url}`);
  if (!content || content?.length === 0) {

View on GitHub (pinned to 3aec848f28)