unslothai/unsloth · critical · HTTPException

Failed to start {log_prefix.lower()}: {scrubbed}

Error message

Failed to start {log_prefix.lower()}: {scrubbed}

What it means

HTTP 500 raised when `spawn()` — launching the download worker subprocess — itself raised an exception (not a worker runtime failure: the process never started). The exception text is secret-scrubbed (hf_token removed), logged with traceback, the job is set to "error" in the registry, and the scrubbed reason is returned. Causes are process/environment level: missing executable, exec format errors, resource limits.

Source

Thrown at studio/backend/hub/services/download_lifecycle.py:1234

            repo_type,
            repo_id,
            getattr(_metadata, "hub_cache", None) if _metadata is not None else None,
            (
                getattr(_metadata, "blob_hashes", frozenset())
                if getattr(_metadata, "variant", None)
                else None
            ),
        )
    try:
        proc = spawn()
    except Exception as e:
        scrubbed = download_registry.scrub_secrets(str(e), hf_token = hf_token)
        logger.error(
            f"Failed to spawn {log_prefix.lower()} worker for {label}: {scrubbed}",
            exc_info = True,
        )
        registry.set_job(key, "error", scrubbed)
        raise HTTPException(
            status_code = 500,
            detail = f"Failed to start {log_prefix.lower()}: {scrubbed}",
        ) from e
    register_worker(
        registry,
        key,
        proc,
        hf_token = hf_token,
        label = label,
        log_prefix = log_prefix,
        logger = logger,
        repo_type = repo_type,
        repo_id = repo_id,
        transport = transport,
        watch_name = watch_name,
        bytes_before = _baseline,
        allow_ambient_token = allow_ambient_token,
    )

View on GitHub (pinned to 203007d190)

Solutions

1. Check the server log line `Failed to spawn ... worker for <label>: <scrubbed>` — it contains the spawn exception class and message.
2. Verify the worker executable/script exists and is executable in the backend environment (which python, ls the worker path).
3. Raise pid/memory limits (cgroup pids.max, ulimit -u) if the host refuses process creation.
4. Reinstall/repair the Studio environment (pip install -e . or rebuild the image) if the venv is broken.

import os, shutil, resource

def can_spawn_worker(python_exe: str) -> bool:
    if not shutil.which(python_exe) and not os.path.exists(python_exe):
        return False
    try:
        pid_limit = resource.getrlimit(resource.RLIMIT_NPROC)[0]
        return pid_limit == resource.RLIM_INFINITY or len(os.listdir('/proc/self/task')) < pid_limit
    except (ValueError, OSError):
        return True  # cannot determine; attempt anyway

try:
    start_download(client, body)
except HTTPStatusError as e:
    if e.response.status_code == 500 and "Failed to start" in e.response.text:
        # spawn-level failure, not a download failure: check job registry shows 'error',
        # inspect server log for the spawn traceback, fix env (PATH/pids), then retry once
        assert_job_in_error_state(client, body.repo_id)
        repair_environment_and_retry_once(body)
    else:
        raise

When it happens

Trigger: The worker interpreter/executable missing from PATH in the backend environment; PosixSpawn/subprocess raising PermissionError; hitting OS process limits (fork bomb protection, cgroup pids.max); exec format mismatch in multi-arch containers; memory too low to fork.

Common situations: Broken venv after a partial upgrade; container images where the worker script path changed; CI/cgroup environments with tight pid limits; Windows permission oddities on the python binary.

Related errors

• {unavailable_reason}
• Git is required to install the pinned {source_name} source
• No export subprocess running
• Failed to send command to subprocess: {exc}
• Export subprocess crashed during wait

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/2ce30812e2a82088. Report an issue: GitHub.