{"record":{"id":"cc8d3858c1d58499","repo":"gocolly/colly","slug":"errrobotstxtblocked","errorCode":"ErrRobotsTxtBlocked","errorMessage":"URL blocked by robots.txt","messagePattern":"URL blocked by robots\\.txt","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"colly.go","lineNumber":236,"sourceCode":"const envVariablePrefix = \"COLLY_\"\n\nvar (\n\t// ErrForbiddenDomain is the error thrown if visiting\n\t// a domain which is not allowed in AllowedDomains\n\tErrForbiddenDomain = errors.New(\"Forbidden domain\")\n\t// ErrMissingURL is the error type for missing URL errors\n\tErrMissingURL = errors.New(\"Missing URL\")\n\t// ErrMaxDepth is the error type for exceeding max depth\n\tErrMaxDepth = errors.New(\"Max depth limit reached\")\n\t// ErrForbiddenURL is the error thrown if visiting\n\t// a URL which is not allowed by URLFilters\n\tErrForbiddenURL = errors.New(\"ForbiddenURL\")\n\n\t// ErrNoURLFiltersMatch is the error thrown if visiting\n\t// a URL which is not allowed by URLFilters\n\tErrNoURLFiltersMatch = errors.New(\"No URLFilters match\")\n\t// ErrRobotsTxtBlocked is the error type for robots.txt errors\n\tErrRobotsTxtBlocked = errors.New(\"URL blocked by robots.txt\")\n\t// ErrNoCookieJar is the error type for missing cookie jar\n\tErrNoCookieJar = errors.New(\"Cookie jar is not available\")\n\t// ErrNoPattern is the error type for LimitRules without patterns\n\tErrNoPattern = errors.New(\"No pattern defined in LimitRule\")\n\t// ErrEmptyProxyURL is the error type for empty Proxy URL list\n\tErrEmptyProxyURL = errors.New(\"Proxy URL list is empty\")\n\t// ErrAbortedAfterHeaders is the error returned when OnResponseHeaders aborts the transfer.\n\tErrAbortedAfterHeaders = errors.New(\"Aborted after receiving response headers\")\n\t// ErrAbortedBeforeRequest is the error returned when OnResponseHeaders aborts the transfer.\n\tErrAbortedBeforeRequest = errors.New(\"Aborted before Do Request\")\n\t// ErrQueueFull is the error returned when the queue is full\n\tErrQueueFull = errors.New(\"Queue MaxSize reached\")\n\t// ErrMaxRequests is the error returned when exceeding max requests\n\tErrMaxRequests = errors.New(\"Max Requests limit reached\")\n\t// ErrRetryBodyUnseekable is the error when retry with not seekable body\n\tErrRetryBodyUnseekable = errors.New(\"Retry Body Unseekable\")\n)\n","sourceCodeStart":218,"sourceCodeEnd":254,"githubUrl":"https://github.com/gocolly/colly/blob/17d1d6ca92bd32a5651f34256bf7a2855c967f65/colly.go#L218-L254","documentation":"ErrRobotsTxtBlocked is returned when the collector's CheckRobotsTxt option is enabled and the target site's robots.txt disallows crawling the requested URL. Colly fetches and honors robots.txt as a politeness/compliance mechanism before making the actual request. The crawl is refused locally — no page request is sent for the disallowed path.","triggerScenarios":"Creating a collector with colly.CheckRobotsTxt() and calling Visit() on a path disallowed by the site's robots.txt (Disallow rules or full-site disallow); visiting a URL under a Disallow prefix discovered mid-crawl via OnHTML link following; robots.txt updated server-side so previously crawlable URLs become blocked.","commonSituations":"Scraping endpoints like /admin, /search, or /api that sites commonly disallow; crawling during a site's heavy-traffic period when a blanket Disallow is active; enabling CheckRobotsTxt on an existing collector without auditing which target paths are blocked.","solutions":["Read the site's robots.txt and restrict your crawl to allowed paths","Remove colly.CheckRobotsTxt() only if you have permission to ignore robots.txt — prefer respecting it","Cache/handle the error in OnError and skip disallowed URL patterns instead of retrying","For allowed paths that still fail, verify the User-Agent matching the robots.txt rules (colly sends its default UA)"],"exampleFix":"// before\nc := colly.NewCollector(colly.CheckRobotsTxt())\nc.Visit(\"https://example.com/admin\") // Disallow: /admin\n// after\nc := colly.NewCollector(colly.CheckRobotsTxt())\nc.Visit(\"https://example.com/public/page\") // allowed by robots.txt","handlingStrategy":"fallback","validationCode":"// Pre-check robots.txt yourself before crawling\nresp, err := http.Get(base + \"/robots.txt\")\n// parse and confirm target path is not Disallowed for your UA","typeGuard":null,"tryCatchPattern":"if err := c.Visit(u); err != nil && errors.Is(err, colly.ErrRobotsTxtBlocked) {\n    log.Printf(\"robots.txt blocks %s, skipping\", u)\n    return nil\n}","preventionTips":["Read robots.txt before planning a crawl with CheckRobotsTxt enabled","Route OnError to skip robots-blocked paths instead of retrying","Only disable CheckRobotsTxt with explicit permission to do so"],"tags":["colly","scraping","robots-txt","compliance","go"],"backgroundTag":"robots-txt-disallow","analyzedSha":"17d1d6ca92bd32a5651f34256bf7a2855c967f65","analyzedAt":"2026-08-30T21:32:31.579Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}