BerriAI/litellm · error · Exception

Internal Error: Cache cannot be empty -…

Error message

Internal Error: Cache cannot be empty - internal_usage_cache={internal_usage_cache}

What it means

Raised when the 'dynamic_rate_limiter' proxy callback is instantiated without an internal usage cache. _PROXY_DynamicRateLimitHandler adjusts limits from observed TPM/RPM usage stored in LiteLLM's internal_usage_cache; the logging initializer passes that cache only when running inside a properly initialized LiteLLM proxy, so SDK-side use (or a proxy state where the cache was never created) makes internal_usage_cache None and this Exception fires.

Solutions

1. Use dynamic_rate_limiter only inside the LiteLLM proxy with a cache configured (litellm_settings: cache: true, ideally redis_url for multi-instance).
2. Upgrade litellm to a matching current version so the proxy passes internal_usage_cache.
3. Remove the callback if you do not actually need usage-based dynamic rate limiting.

Example fix

# before (SDK)
import litellm
litellm.callbacks = ['dynamic_rate_limiter']  # Exception: Internal Error: Cache cannot be empty

# after: run it on the proxy with a cache
# config.yaml
# litellm_settings:
#   cache: true
#   callbacks: ['dynamic_rate_limiter']

# dynamic_rate_limiter is proxy-only: verify you are on the proxy with a cache before enabling it
import litellm
cache_ok = getattr(litellm, 'cache', None) is not None or os.environ.get('REDIS_HOST')
if not cache_ok:
    raise RuntimeError('enable litellm_settings.cache (and redis) before using dynamic_rate_limiter')

try:
    litellm.completion(model='gpt-4o', messages=m)
except Exception as e:
    if 'Internal Error: Cache cannot be empty' in str(e):
        # remove 'dynamic_rate_limiter' from callbacks or configure litellm_settings.cache on the proxy
        raise
    raise

When it happens

Trigger: Setting litellm.callbacks = ['dynamic_rate_limiter'] in plain SDK code (no proxy); enabling the callback before proxy cache initialization; running an older/newer proxy where internal_usage_cache is not wired into initialize's arguments.

Common situations: Reusing proxy-style config (litellm_settings.callbacks) in SDK scripts; minimal proxy deployments without any cache configured (no litellm_settings.cache block / redis_url); version drift between litellm core and proxy extras.

Related errors

• Callback param ' ' (from ) contains an 'os.environ/'…
• litellm_settings.callbacks entry
• litellm_settings.callbacks entry
• MissingTrying to use Llama GuardMissing litellm-enterprise…
• No active custom logger found for callback name

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/b5f643f8f6671d62. Report an issue: GitHub.

Appendix: source

Thrown at litellm/litellm_core_utils/litellm_logging.py:4141

            )
            for callback in _in_memory_loggers:
                # Use exact type check to avoid matching ArizePhoenixLogger (subclass)
                if type(callback) is OpenTelemetry:
                    return callback
            _otel_logger = OpenTelemetry(config=otel_config)
            _in_memory_loggers.append(_otel_logger)
            return _otel_logger
        elif logging_integration == "dynamic_rate_limiter":
            from litellm.proxy.hooks.dynamic_rate_limiter import (
                _PROXY_DynamicRateLimitHandler,
            )

            for callback in _in_memory_loggers:
                if isinstance(callback, _PROXY_DynamicRateLimitHandler):
                    return callback

            if internal_usage_cache is None:
                raise Exception(f"Internal Error: Cache cannot be empty - internal_usage_cache={internal_usage_cache}")

            dynamic_rate_limiter_obj: Final = _PROXY_DynamicRateLimitHandler(internal_usage_cache=internal_usage_cache)

            if llm_router is not None and isinstance(llm_router, litellm.Router):
                dynamic_rate_limiter_obj.update_variables(llm_router=llm_router)
            _in_memory_loggers.append(dynamic_rate_limiter_obj)
            return dynamic_rate_limiter_obj
        elif logging_integration == "dynamic_rate_limiter_v3":
            from litellm.proxy.hooks.dynamic_rate_limiter_v3 import (
                _PROXY_DynamicRateLimitHandlerV3,
            )

            for callback in _in_memory_loggers:
                if isinstance(callback, _PROXY_DynamicRateLimitHandlerV3):
                    return callback

            if internal_usage_cache is None:
                raise Exception(f"Internal Error: Cache cannot be empty - internal_usage_cache={internal_usage_cache}")

View on GitHub (pinned to 77b7c6c40c)