{"record":{"id":"8ed897b3a90d5952","repo":"GoogleChrome/lighthouse","slug":"unknown-directive","errorCode":null,"errorMessage":"Unknown directive","messagePattern":"Unknown directive","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"warning","filePath":"core/audits/seo/robots-txt.js","lineNumber":71,"sourceCode":"  /**\n   * @description Explanatory message stating that there was a failure in an audit caused by Lighthouse not being able to download the robots.txt file for the site.  Note: \"robots.txt\" is a canonical filename and should not be translated.\n   * @example {Timed out fetching resource} error\n   * */\n  explanationWithError: 'Fetch of robots.txt failed: {error}',\n  /** Explanatory message stating that there was a failure in an audit caused by Lighthouse not being able to download the robots.txt file for the site.  Note: \"robots.txt\" is a canonical filename and should not be translated. */\n  explanation: 'Fetch of robots.txt failed',\n};\n\nconst str_ = i18n.createIcuMessageFn(import.meta.url, UIStrings);\n\n/**\n * @param {string} directiveName\n * @param {string} directiveValue\n * @throws will throw an exception if given directive is invalid\n */\nfunction verifyDirective(directiveName, directiveValue) {\n  if (!DIRECTIVE_SAFELIST.has(directiveName)) {\n    throw new Error('Unknown directive');\n  }\n\n  if (directiveName === DIRECTIVE_SITEMAP) {\n    let sitemapUrl;\n\n    try {\n      sitemapUrl = new URL(directiveValue);\n    } catch (e) {\n      throw new Error('Invalid sitemap URL');\n    }\n\n    if (!SITEMAP_VALID_PROTOCOLS.has(sitemapUrl.protocol)) {\n      throw new Error('Invalid sitemap URL protocol');\n    }\n  }\n\n  if (directiveName === DIRECTIVE_USER_AGENT && !directiveValue) {\n    throw new Error('No user-agent specified');","sourceCodeStart":53,"sourceCodeEnd":89,"githubUrl":"https://github.com/GoogleChrome/lighthouse/blob/9515cd4e58ebed69f78742d932b501c2cab8ad8f/core/audits/seo/robots-txt.js#L53-L89","documentation":"Lighthouse's robots.txt audit validates the site's robots.txt against a safelist of recognized directives: user-agent, disallow, allow, sitemap, crawl-delay, clean-param, host, request-rate, visit-time, noindex, and content-signal. The verifyDirective function throws this error when a parsed directive name is not in the DIRECTIVE_SAFELIST set. The error is caught internally by validateRobots and surfaced as a validation error item in the audit results rather than propagated as a thrown exception.","triggerScenarios":"The audited site's robots.txt contains a directive name that is not in Lighthouse's safelist. For example, a line like 'acap-crawler: *' or 'some-custom-directive: value' would trigger this when parsed. The parser splits on ':', lowercases the directive name, and checks membership in DIRECTIVE_SAFELIST.","commonSituations":"Site uses proprietary or non-standard robots.txt directives not in Lighthouse's safelist; typos in standard directive names (e.g., 'usr-agent' instead of 'user-agent'); copy-paste from documentation that uses a non-standard extension; CDN or framework injecting non-standard directives.","solutions":["Review the site's robots.txt and identify the directive flagged as unknown","Remove or comment out the unrecognized directive if it serves no purpose","If the directive is a legitimate extension, verify it is spelled correctly and matches a safelisted name","If the directive must remain, this audit will report it as an error — weight the SEO impact accordingly"],"exampleFix":"# before (robots.txt)\nUser-agent: *\nDisallow: /private\nacap-crawler: *\n\n# after (robots.txt)\nUser-agent: *\nDisallow: /private","handlingStrategy":"validation","validationCode":"// Validate robots.txt directives before deployment\nconst SAFE_DIRECTIVES = new Set(['user-agent','disallow','allow','sitemap','crawl-delay','clean-param','host','request-rate','visit-time','noindex','content-signal']);\nfunction validateRobotsDirectives(content) {\n  const errors = [];\n  for (const line of content.split(/\\r\\n|\\r|\\n/)) {\n    const stripped = line.split('#')[0].trim();\n    if (!stripped) continue;\n    const colonIdx = stripped.indexOf(':');\n    if (colonIdx === -1) { errors.push(`Syntax error: ${line}`); continue; }\n    const name = stripped.slice(0, colonIdx).trim().toLowerCase();\n    if (!SAFE_DIRECTIVES.has(name)) errors.push(`Unknown directive: ${name}`);\n  }\n  return errors;\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate robots.txt with Google Search Console or an online validator before deployment","Keep robots.txt to standard, widely-supported directives","Remove or comment out proprietary directives that serve no crawler purpose","Run a Lighthouse SEO audit in CI to catch robots.txt regressions"],"tags":["seo","robots-txt","validation"],"backgroundTag":null,"analyzedSha":"9515cd4e58ebed69f78742d932b501c2cab8ad8f","analyzedAt":"2026-08-13T06:28:10.346Z","schemaVersion":2},"datasetVersion":"2026-08-13T09:17:06.757Z"}