{"record":{"id":"72a21fbac82250c5","repo":"Mintplex-Labs/anything-llm","slug":"url-is-required-for-web-scraping","errorCode":null,"errorMessage":"URL is required for web scraping","messagePattern":"URL is required for web scraping","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"server/utils/agentFlows/executors/web-scraping.js","lineNumber":20,"sourceCode":" * Execute a web scraping flow step\n * @param {Object} config Flow step configuration\n * @param {Object} context Execution context with introspect function\n * @returns {Promise<string>} Scraped content\n */\nasync function executeWebScraping(config, context) {\n  const { CollectorApi } = require(\"../../collectorApi\");\n  const { TokenManager } = require(\"../../helpers/tiktoken\");\n  const Provider = require(\"../../agents/aibitat/providers/ai-provider\");\n  const { summarizeContent } = require(\"../../agents/aibitat/utils/summarize\");\n\n  const { url, captureAs = \"text\", enableSummarization = true } = config;\n  const { introspect, logger, aibitat } = context;\n  logger(\n    `\\x1b[43m[AgentFlowToolExecutor]\\x1b[0m - executing Web Scraping block`\n  );\n\n  if (!url) {\n    throw new Error(\"URL is required for web scraping\");\n  }\n\n  const captureMode = captureAs === \"querySelector\" ? \"html\" : captureAs;\n  introspect(`Scraping the content of ${url} as ${captureAs}`);\n  const { success, content } = await new CollectorApi()\n    .getLinkContent(url, captureMode)\n    .then((res) => {\n      if (captureAs !== \"querySelector\") return res;\n      return parseHTMLwithSelector(res.content, config.querySelector, context);\n    });\n\n  if (!success) {\n    introspect(`Could not scrape ${url}. Cannot use this page's content.`);\n    throw new Error(\"URL could not be scraped and no content was found.\");\n  }\n\n  introspect(`Successfully scraped content from ${url}`);\n  if (!content || content?.length === 0) {","sourceCodeStart":2,"sourceCodeEnd":38,"githubUrl":"https://github.com/Mintplex-Labs/anything-llm/blob/526360e320da9d1b36074be5ed64fe76e5bbfbbd/server/utils/agentFlows/executors/web-scraping.js#L2-L38","documentation":"Thrown at the top of executeWebScraping() when config.url is falsy after destructuring. The URL is mandatory before any collector call is attempted. This is a pure input-validation guard.","triggerScenarios":"A webScraping step whose config has no `url` field, an empty url string, or a url built from an unresolved variable template like ${pageUrl} where the variable was never set (replaceVariables leaves unmatched ${...} as-is, so the url would be non-empty in that case, but a missing field entirely triggers this).","commonSituations":"Block created without filling the URL; the url field references a variable that does not exist in the flow's variable map; the start block did not declare the variable used in the url.","solutions":["Set a non-empty url in the webScraping step config.","If using a variable like ${pageUrl}, ensure the start block declares it and a value is supplied at execution time.","Verify the variable name spelling matches between the start block and the url template."],"exampleFix":"// before - no url supplied\n{\"type\":\"webScraping\",\"config\":{\"captureAs\":\"text\"}}\n// after - url provided\n{\"type\":\"webScraping\",\"config\":{\"url\":\"https://example.com\",\"captureAs\":\"text\"}}","handlingStrategy":"validation","validationCode":"function validateWebScrapingConfig(config) {\n  if (!config.url || typeof config.url !== \"string\")\n    throw new Error(\"webScraping step requires a non-empty url\");\n}","typeGuard":"const hasScrapeUrl = (config) => typeof config?.url === \"string\" && config.url.trim().length > 0;","tryCatchPattern":null,"preventionTips":["Always fill the url field when creating a webScraping block.","When referencing a variable in url, declare it in the start block and supply a value at run time.","Double-check variable name spelling between the start block and the url template."],"tags":["web-scraping","validation","agent-flows"],"backgroundTag":null,"analyzedSha":"526360e320da9d1b36074be5ed64fe76e5bbfbbd","analyzedAt":"2026-08-13T01:45:47.170Z","schemaVersion":2},"datasetVersion":"2026-08-13T04:17:16.726Z"}