infiniflow/ragflow · warning · RuntimeError

Tenki rate limited, please retry: {exc}

Error message

Tenki rate limited, please retry: {exc}

What it means

Raised when client.create() throws errors.RateLimitedError: the Tenki API throttled the sandbox-creation request. Unlike quota errors this is transient — the message explicitly says 'please retry' — and retrying after a backoff typically succeeds.

Source

Thrown at agent/sandbox/providers/tenki.py:149

            "allow_outbound": self.allow_outbound,
            "max_duration": self.max_lifetime,
            "metadata": {"source": "ragflow"},
        }
        if self.image:
            create_kwargs["image"] = self.image
        if self.cpu_cores > 0:
            create_kwargs["cpu_cores"] = self.cpu_cores
        if self.memory_mb > 0:
            create_kwargs["memory_mb"] = self.memory_mb
        if self.disk_size_gb > 0:
            create_kwargs["disk_size_gb"] = self.disk_size_gb

        try:
            sandbox = self._client.create(**create_kwargs)
        except errors.QuotaExceededError as exc:
            raise RuntimeError(f"Tenki quota exceeded: {exc}") from exc
        except errors.RateLimitedError as exc:
            raise RuntimeError(f"Tenki rate limited, please retry: {exc}") from exc
        except errors.UnauthorizedError as exc:
            raise SandboxProviderConfigError("Tenki authentication failed: check the API key.") from exc
        except Exception as exc:
            # Satisfy the base contract: any other SDK failure becomes RuntimeError.
            raise RuntimeError(f"Failed to create Tenki sandbox: {exc}") from exc

        remote_work_dir = posixpath.join(SANDBOX_HOME, f"ragflow-codeexec-{uuid.uuid4().hex}")
        try:
            result = sandbox.exec(
                "mkdir",
                "-p",
                posixpath.join(remote_work_dir, "artifacts"),
                timeout=min(self.timeout, 10),
            )
            if result.exit_code != 0:
                raise RuntimeError(f"Failed to create sandbox workspace: {result.stderr_text or 'unknown error'}")
        except Exception:
            self._safe_terminate(sandbox)

View on GitHub (pinned to 554fb1133a)

Solutions

1. Retry create_instance() with exponential backoff and jitter (this error is explicitly retryable).
2. Add client-side throttling (semaphore or token bucket) in front of create_instance().
3. Cache/reuse a sandbox for multiple execute_code() runs instead of creating one per request.

Example fix

# before
sandbox = provider.create_instance()

# after
for attempt in range(5):
    try:
        sandbox = provider.create_instance()
        break
    except RuntimeError as exc:
        if "rate limited" not in str(exc) or attempt == 4:
            raise
        time.sleep(2 ** attempt + random.random())

def create_with_backoff(provider, template, attempts=5):
    for i in range(attempts):
        try:
            return provider.create_instance(template)
        except RuntimeError as exc:
            if "rate limited" not in str(exc) or i == attempts - 1:
                raise
            time.sleep((2 ** i) + random.random())

When it happens

Trigger: Bursty create_instance() calls exceeding the Tenki API's requests-per-second limit, e.g. a fan-out of agents each provisioning a sandbox at the same moment.

Common situations: Load tests or batch ingestion jobs that create many sandboxes in a tight loop; shared API keys across multiple RAGFlow deployments hitting the same rate bucket.

Related errors

• UCloud Agent Sandbox rate limited, please retry: {exc}
• Invalid Tenki provider configuration.
• Provider not initialized. Call initialize() first.
• Tenki quota exceeded: {exc}
• Failed to create Tenki sandbox: {exc}

AI-assisted analysis of infiniflow/ragflow@554fb1133a (2026-08-15). Data as JSON: /api/errors/de1cf53c4e12655c. Report an issue: GitHub.