{"record":{"id":"2ce30812e2a82088","repo":"unslothai/unsloth","slug":"failed-to-start-log-prefix-lower-scrubbed","errorCode":null,"errorMessage":"Failed to start {log_prefix.lower()}: {scrubbed}","messagePattern":"Failed to start (.+?): (.+?)","errorType":"http","errorClass":"HTTPException","httpStatus":500,"severity":"critical","filePath":"studio/backend/hub/services/download_lifecycle.py","lineNumber":1234,"sourceCode":"            repo_type,\n            repo_id,\n            getattr(_metadata, \"hub_cache\", None) if _metadata is not None else None,\n            (\n                getattr(_metadata, \"blob_hashes\", frozenset())\n                if getattr(_metadata, \"variant\", None)\n                else None\n            ),\n        )\n    try:\n        proc = spawn()\n    except Exception as e:\n        scrubbed = download_registry.scrub_secrets(str(e), hf_token = hf_token)\n        logger.error(\n            f\"Failed to spawn {log_prefix.lower()} worker for {label}: {scrubbed}\",\n            exc_info = True,\n        )\n        registry.set_job(key, \"error\", scrubbed)\n        raise HTTPException(\n            status_code = 500,\n            detail = f\"Failed to start {log_prefix.lower()}: {scrubbed}\",\n        ) from e\n    register_worker(\n        registry,\n        key,\n        proc,\n        hf_token = hf_token,\n        label = label,\n        log_prefix = log_prefix,\n        logger = logger,\n        repo_type = repo_type,\n        repo_id = repo_id,\n        transport = transport,\n        watch_name = watch_name,\n        bytes_before = _baseline,\n        allow_ambient_token = allow_ambient_token,\n    )","sourceCodeStart":1216,"sourceCodeEnd":1252,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/hub/services/download_lifecycle.py#L1216-L1252","documentation":"HTTP 500 raised when `spawn()` — launching the download worker subprocess — itself raised an exception (not a worker runtime failure: the process never started). The exception text is secret-scrubbed (hf_token removed), logged with traceback, the job is set to \"error\" in the registry, and the scrubbed reason is returned. Causes are process/environment level: missing executable, exec format errors, resource limits.","triggerScenarios":"The worker interpreter/executable missing from PATH in the backend environment; PosixSpawn/subprocess raising PermissionError; hitting OS process limits (fork bomb protection, cgroup pids.max); exec format mismatch in multi-arch containers; memory too low to fork.","commonSituations":"Broken venv after a partial upgrade; container images where the worker script path changed; CI/cgroup environments with tight pid limits; Windows permission oddities on the python binary.","solutions":["Check the server log line `Failed to spawn ... worker for <label>: <scrubbed>` — it contains the spawn exception class and message.","Verify the worker executable/script exists and is executable in the backend environment (which python, ls the worker path).","Raise pid/memory limits (cgroup pids.max, ulimit -u) if the host refuses process creation.","Reinstall/repair the Studio environment (pip install -e . or rebuild the image) if the venv is broken."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import os, shutil, resource\n\ndef can_spawn_worker(python_exe: str) -> bool:\n    if not shutil.which(python_exe) and not os.path.exists(python_exe):\n        return False\n    try:\n        pid_limit = resource.getrlimit(resource.RLIMIT_NPROC)[0]\n        return pid_limit == resource.RLIM_INFINITY or len(os.listdir('/proc/self/task')) < pid_limit\n    except (ValueError, OSError):\n        return True  # cannot determine; attempt anyway","typeGuard":null,"tryCatchPattern":"try:\n    start_download(client, body)\nexcept HTTPStatusError as e:\n    if e.response.status_code == 500 and \"Failed to start\" in e.response.text:\n        # spawn-level failure, not a download failure: check job registry shows 'error',\n        # inspect server log for the spawn traceback, fix env (PATH/pids), then retry once\n        assert_job_in_error_state(client, body.repo_id)\n        repair_environment_and_retry_once(body)\n    else:\n        raise","preventionTips":["Health-check the worker executable on backend startup, not at first download.","Set adequate cgroup pid/memory limits for the Studio container.","Pin dependency layout (worker script path) across upgrades to avoid broken spawn paths."],"tags":["subprocess","spawn","environment","http-500","download"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}