{"record":{"id":"41bcb853be414386","repo":"GoogleChrome/lighthouse","slug":"no-user-agent-specified","errorCode":null,"errorMessage":"No user-agent specified","messagePattern":"No user-agent specified","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"warning","filePath":"core/audits/seo/robots-txt.js","lineNumber":89,"sourceCode":"    throw new Error('Unknown directive');\n  }\n\n  if (directiveName === DIRECTIVE_SITEMAP) {\n    let sitemapUrl;\n\n    try {\n      sitemapUrl = new URL(directiveValue);\n    } catch (e) {\n      throw new Error('Invalid sitemap URL');\n    }\n\n    if (!SITEMAP_VALID_PROTOCOLS.has(sitemapUrl.protocol)) {\n      throw new Error('Invalid sitemap URL protocol');\n    }\n  }\n\n  if (directiveName === DIRECTIVE_USER_AGENT && !directiveValue) {\n    throw new Error('No user-agent specified');\n  }\n\n  if (directiveName === DIRECTIVE_ALLOW || directiveName === DIRECTIVE_DISALLOW) {\n    if (directiveValue !== '' && directiveValue[0] !== '/' && directiveValue[0] !== '*') {\n      throw new Error('Pattern should either be empty, start with \"/\" or \"*\"');\n    }\n\n    const dollarIndex = directiveValue.indexOf('$');\n\n    if (dollarIndex !== -1 && dollarIndex !== directiveValue.length - 1) {\n      throw new Error('\"$\" should only be used at the end of the pattern');\n    }\n  }\n}\n\n/**\n * @param {string} line single line from a robots.txt file\n * @throws will throw an exception if given line has errors","sourceCodeStart":71,"sourceCodeEnd":107,"githubUrl":"https://github.com/GoogleChrome/lighthouse/blob/9515cd4e58ebed69f78742d932b501c2cab8ad8f/core/audits/seo/robots-txt.js#L71-L107","documentation":"Thrown by verifyDirective() when a 'User-agent:' line in robots.txt has no value after the colon. Lighthouse's robots.txt audit parses each line and validates directive syntax; an empty user-agent field is invalid per the robots exclusion standard. The error is caught internally by validateRobots() (line 156) and surfaced as an audit failure item, not an unhandled crash.","triggerScenarios":"A robots.txt line containing 'User-agent:' with nothing after the colon, or only whitespace/comments after the colon (e.g. 'User-agent: # comment'). The parser trims the value and finds it empty, so the check `!directiveValue` passes and throws.","commonSituations":"Hand-edited robots.txt where a developer forgot to specify the bot name. Migrating robots.txt rules and accidentally leaving a directive value blank. A CMS or plugin that generates malformed robots.txt output.","solutions":["Add a user-agent value after the colon, e.g. 'User-agent: *' for all crawlers","If targeting a specific crawler, use its name, e.g. 'User-agent: Googlebot'","Remove the bare 'User-agent:' line if it was unintentional"],"exampleFix":"// before\nUser-agent:\nDisallow: /private\n\n// after\nUser-agent: *\nDisallow: /private","handlingStrategy":"validation","validationCode":"// Validate robots.txt content before parsing\nfunction validateUserAgentLines(content) {\n  const lines = content.split(/\\r\\n|\\r|\\n/);\n  const errors = [];\n  lines.forEach((line, idx) => {\n    const trimmed = line.split('#')[0].trim();\n    if (/^user-agent\\s*:\\s*$/i.test(trimmed)) {\n      errors.push(`Line ${idx + 1}: User-agent has no value`);\n    }\n  });\n  return errors;\n}","typeGuard":null,"tryCatchPattern":"// Lighthouse already catches this internally in validateRobots().\n// If calling parseLine() directly:\ntry {\n  parseLine(line);\n} catch (e) {\n  // e.message === 'No user-agent specified'\n  console.warn(`robots.txt parse error: ${e.message}`);\n}","preventionTips":["Always specify a value after 'User-agent:' — use '*' for all crawlers","Validate robots.txt with an online robots.txt tester before deploying","Use a linter or validator plugin in your CI pipeline for robots.txt"],"tags":["robots-txt","seo","validation","parsing"],"backgroundTag":null,"analyzedSha":"9515cd4e58ebed69f78742d932b501c2cab8ad8f","analyzedAt":"2026-08-13T06:28:10.346Z","schemaVersion":2},"datasetVersion":"2026-08-13T09:17:06.757Z"}