GoogleChrome/lighthouse · warning · Error

Unknown directive

Error message

Unknown directive

What it means

Lighthouse's robots.txt audit validates the site's robots.txt against a safelist of recognized directives: user-agent, disallow, allow, sitemap, crawl-delay, clean-param, host, request-rate, visit-time, noindex, and content-signal. The verifyDirective function throws this error when a parsed directive name is not in the DIRECTIVE_SAFELIST set. The error is caught internally by validateRobots and surfaced as a validation error item in the audit results rather than propagated as a thrown exception.

Source

Thrown at core/audits/seo/robots-txt.js:71

  /**
   * @description Explanatory message stating that there was a failure in an audit caused by Lighthouse not being able to download the robots.txt file for the site.  Note: "robots.txt" is a canonical filename and should not be translated.
   * @example {Timed out fetching resource} error
   * */
  explanationWithError: 'Fetch of robots.txt failed: {error}',
  /** Explanatory message stating that there was a failure in an audit caused by Lighthouse not being able to download the robots.txt file for the site.  Note: "robots.txt" is a canonical filename and should not be translated. */
  explanation: 'Fetch of robots.txt failed',
};

const str_ = i18n.createIcuMessageFn(import.meta.url, UIStrings);

/**
 * @param {string} directiveName
 * @param {string} directiveValue
 * @throws will throw an exception if given directive is invalid
 */
function verifyDirective(directiveName, directiveValue) {
  if (!DIRECTIVE_SAFELIST.has(directiveName)) {
    throw new Error('Unknown directive');
  }

  if (directiveName === DIRECTIVE_SITEMAP) {
    let sitemapUrl;

    try {
      sitemapUrl = new URL(directiveValue);
    } catch (e) {
      throw new Error('Invalid sitemap URL');
    }

    if (!SITEMAP_VALID_PROTOCOLS.has(sitemapUrl.protocol)) {
      throw new Error('Invalid sitemap URL protocol');
    }
  }

  if (directiveName === DIRECTIVE_USER_AGENT && !directiveValue) {
    throw new Error('No user-agent specified');

View on GitHub (pinned to 9515cd4e58)

Solutions

1. Review the site's robots.txt and identify the directive flagged as unknown
2. Remove or comment out the unrecognized directive if it serves no purpose
3. If the directive is a legitimate extension, verify it is spelled correctly and matches a safelisted name
4. If the directive must remain, this audit will report it as an error — weight the SEO impact accordingly

Example fix

# before (robots.txt)
User-agent: *
Disallow: /private
acap-crawler: *

# after (robots.txt)
User-agent: *
Disallow: /private

// Validate robots.txt directives before deployment
const SAFE_DIRECTIVES = new Set(['user-agent','disallow','allow','sitemap','crawl-delay','clean-param','host','request-rate','visit-time','noindex','content-signal']);
function validateRobotsDirectives(content) {
  const errors = [];
  for (const line of content.split(/\r\n|\r|\n/)) {
    const stripped = line.split('#')[0].trim();
    if (!stripped) continue;
    const colonIdx = stripped.indexOf(':');
    if (colonIdx === -1) { errors.push(`Syntax error: ${line}`); continue; }
    const name = stripped.slice(0, colonIdx).trim().toLowerCase();
    if (!SAFE_DIRECTIVES.has(name)) errors.push(`Unknown directive: ${name}`);
  }
  return errors;
}

When it happens

Trigger: The audited site's robots.txt contains a directive name that is not in Lighthouse's safelist. For example, a line like 'acap-crawler: *' or 'some-custom-directive: value' would trigger this when parsed. The parser splits on ':', lowercases the directive name, and checks membership in DIRECTIVE_SAFELIST.

Common situations: Site uses proprietary or non-standard robots.txt directives not in Lighthouse's safelist; typos in standard directive names (e.g., 'usr-agent' instead of 'user-agent'); copy-paste from documentation that uses a non-standard extension; CDN or framework injecting non-standard directives.

Related errors

• Invalid sitemap URL
• Invalid sitemap URL protocol
• No user-agent specified
• Pattern should either be empty, start with "/" or "*"
• "$" should only be used at the end of the pattern

AI-assisted analysis of GoogleChrome/lighthouse@9515cd4e58 (2026-08-13). Data as JSON: /api/errors/8ed897b3a90d5952. Report an issue: GitHub.