GoogleChrome/lighthouse · warning · Error

No user-agent specified

Error message

No user-agent specified

What it means

Thrown by verifyDirective() when a 'User-agent:' line in robots.txt has no value after the colon. Lighthouse's robots.txt audit parses each line and validates directive syntax; an empty user-agent field is invalid per the robots exclusion standard. The error is caught internally by validateRobots() (line 156) and surfaced as an audit failure item, not an unhandled crash.

Source

Thrown at core/audits/seo/robots-txt.js:89

    throw new Error('Unknown directive');
  }

  if (directiveName === DIRECTIVE_SITEMAP) {
    let sitemapUrl;

    try {
      sitemapUrl = new URL(directiveValue);
    } catch (e) {
      throw new Error('Invalid sitemap URL');
    }

    if (!SITEMAP_VALID_PROTOCOLS.has(sitemapUrl.protocol)) {
      throw new Error('Invalid sitemap URL protocol');
    }
  }

  if (directiveName === DIRECTIVE_USER_AGENT && !directiveValue) {
    throw new Error('No user-agent specified');
  }

  if (directiveName === DIRECTIVE_ALLOW || directiveName === DIRECTIVE_DISALLOW) {
    if (directiveValue !== '' && directiveValue[0] !== '/' && directiveValue[0] !== '*') {
      throw new Error('Pattern should either be empty, start with "/" or "*"');
    }

    const dollarIndex = directiveValue.indexOf('$');

    if (dollarIndex !== -1 && dollarIndex !== directiveValue.length - 1) {
      throw new Error('"$" should only be used at the end of the pattern');
    }
  }
}

/**
 * @param {string} line single line from a robots.txt file
 * @throws will throw an exception if given line has errors

View on GitHub (pinned to 9515cd4e58)

Solutions

1. Add a user-agent value after the colon, e.g. 'User-agent: *' for all crawlers
2. If targeting a specific crawler, use its name, e.g. 'User-agent: Googlebot'
3. Remove the bare 'User-agent:' line if it was unintentional

Example fix

// before
User-agent:
Disallow: /private

// after
User-agent: *
Disallow: /private

// Validate robots.txt content before parsing
function validateUserAgentLines(content) {
  const lines = content.split(/\r\n|\r|\n/);
  const errors = [];
  lines.forEach((line, idx) => {
    const trimmed = line.split('#')[0].trim();
    if (/^user-agent\s*:\s*$/i.test(trimmed)) {
      errors.push(`Line ${idx + 1}: User-agent has no value`);
    }
  });
  return errors;
}

// Lighthouse already catches this internally in validateRobots().
// If calling parseLine() directly:
try {
  parseLine(line);
} catch (e) {
  // e.message === 'No user-agent specified'
  console.warn(`robots.txt parse error: ${e.message}`);
}

When it happens

Trigger: A robots.txt line containing 'User-agent:' with nothing after the colon, or only whitespace/comments after the colon (e.g. 'User-agent: # comment'). The parser trims the value and finds it empty, so the check `!directiveValue` passes and throws.

Common situations: Hand-edited robots.txt where a developer forgot to specify the bot name. Migrating robots.txt rules and accidentally leaving a directive value blank. A CMS or plugin that generates malformed robots.txt output.

Related errors

• Pattern should either be empty, start with "/" or "*"
• "$" should only be used at the end of the pattern
• Syntax not understood
• Unknown directive
• Invalid sitemap URL

AI-assisted analysis of GoogleChrome/lighthouse@9515cd4e58 (2026-08-13). Data as JSON: /api/errors/41bcb853be414386. Report an issue: GitHub.