{"record":{"id":"634e3aace9902946","repo":"BerriAI/litellm","slug":"blocking-web-crawlers-is-an-enterprise-feature-yo","errorCode":null,"errorMessage":"Blocking web crawlers is an enterprise feature. You must be a LiteLLM Enterprise user to use this feature. If you have a license please set `LITELLM_LICENSE` in your env. Get a 7 day trial key here: https://www.litellm.ai/enterprise#trial. \nPricing: https://www.litellm.ai/#pricing","messagePattern":"Blocking web crawlers is an enterprise feature\\. You must be a LiteLLM Enterprise user to use this feature\\. If you have a license please set `LITELLM_LICENSE` in your env\\. Get a 7 day trial key here: https://www\\.litellm\\.ai/enterprise#trial\\. \nPricing: https://www\\.litellm\\.ai/#pricing","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"enterprise/litellm_enterprise/proxy/utils.py","lineNumber":31,"sourceCode":"    general_settings:\n      block_robots: true\n    ```\n    \"\"\"\n    from litellm.proxy.proxy_server import (\n        CommonProxyErrors,\n        general_settings,\n        premium_user,\n    )\n\n    _block_robots: Union[bool, str] = general_settings.get(\"block_robots\", False)\n    block_robots: Optional[bool] = None\n    if isinstance(_block_robots, bool):\n        block_robots = _block_robots\n    elif isinstance(_block_robots, str):\n        block_robots = str_to_bool(_block_robots)\n    if block_robots is True:\n        if premium_user is not True:\n            raise ValueError(\n                f\"Blocking web crawlers is an enterprise feature. {CommonProxyErrors.not_premium_user.value}\"\n            )\n        return True\n    return False\n","sourceCodeStart":13,"sourceCodeEnd":36,"githubUrl":"https://github.com/BerriAI/litellm/blob/77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8/enterprise/litellm_enterprise/proxy/utils.py#L13-L36","documentation":"Raised by the LiteLLM Proxy when general_settings.block_robots is enabled (true, or a truthy string coerced via str_to_bool) but no valid LiteLLM Enterprise license is present (premium_user is not True). It is a hard startup/config gating error: robot blocking for the proxy UI/docs is an enterprise-only feature. The appended CommonProxyErrors.not_premium_user text tells you to set the LITELLM_LICENSE env var or get a trial key.","triggerScenarios":"Setting `general_settings: block_robots: true` (or \"true\") in the proxy config.yaml without LITELLM_LICENSE set; passing a block_robots setting while running the OSS proxy; running a container where LITELLM_LICENSE was not propagated; using an expired/invalid license key so premium_user resolves False.","commonSituations":"Self-hosting the proxy publicly and wanting to hide it from search crawlers without a subscription; license expired after a 7-day trial; docker-compose/k8s secret not mounted so the env var is missing at config load.","solutions":["Get an enterprise trial/license key and set the LITELLM_LICENSE environment variable, then restart the proxy","If you do not have a license, remove `block_robots` from general_settings (the proxy then starts without robot blocking)","Verify the license actually reaches the process: print os.environ.get('LITELLM_LICENSE') in the proxy container / check the premium_user resolution logs","If you have a license and still see this, upgrade litellm and litellm-enterprise to matching versions so license validation succeeds"],"exampleFix":"# before (config.yaml)\ngeneral_settings:\n  block_robots: true   # raises ValueError, no license\n\n# after (option 1: license)\nexport LITELLM_LICENSE=\"sk-litellm-enterprise-...\"\n# config.yaml unchanged\n\n# after (option 2: no license)\ngeneral_settings: {}   # block_robots removed","handlingStrategy":"validation","validationCode":"import os\n\nhas_license = bool(os.environ.get(\"LITELLM_LICENSE\"))\nwants_block_robots = True  # from your config build step\n\nif wants_block_robots and not has_license:\n    raise SystemExit(\n        \"block_robots requires LiteLLM Enterprise: set LITELLM_LICENSE or drop the setting\"\n    )","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat block_robots as an enterprise-only flag in config templates; gate it behind a license check in your config pipeline","Add a CI config-lint that fails if block_robots is set without LITELLM_LICENSE present in the deploy environment","Document which env vars each deployment template needs, and assert required vars at container entrypoint"],"tags":["enterprise","licensing","proxy-config","startup","block-robots"],"backgroundTag":"enterprise-license-required","analyzedSha":"77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8","analyzedAt":"2026-08-18T11:44:31.656Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}