{"record":{"id":"38efab7d2c64d6b3","repo":"unslothai/unsloth","slug":"stt-model-loading-was-cancelled-so-training-could-38efab","errorCode":null,"errorMessage":"STT model loading was cancelled so training could start.","messagePattern":"STT model loading was cancelled so training could start\\.","errorType":"http","errorClass":"SttLoadCancelledError","httpStatus":409,"severity":"info","filePath":"studio/backend/core/inference/stt_transformers_worker.py","lineNumber":97,"sourceCode":"def _ensure_backend_on_path() -> None:\n    if _BACKEND_PATH not in sys.path:\n        sys.path.insert(0, _BACKEND_PATH)\n\n\nclass _CancelCriteria:\n    \"\"\"Stops generation as soon as the parent sets the shared cancel event.\"\"\"\n\n    def __init__(self, cancel_event) -> None:\n        self._cancel_event = cancel_event\n\n    def __call__(self, *_args, **_kwargs) -> bool:\n        return self._cancel_event.is_set()\n\n\ndef _raise_if_cancelled(cancel_event) -> None:\n    from core.inference.stt_sidecar import SttLoadCancelledError\n    if cancel_event is not None and cancel_event.is_set():\n        raise SttLoadCancelledError(\"STT model loading was cancelled so training could start.\")\n\n\ndef load_whisper(\n    snapshot_path: str,\n    device: str,\n    dtype_name: str,\n    cancel_event = None,\n) -> tuple:\n    \"\"\"Load a Whisper model + processor from the local Hub cache. Child side.\n\n    local_files_only keeps the Model Hub the only download path; a cache miss\n    raises so the parent can surface SttModelNotDownloadedError.\n    \"\"\"\n    import torch\n    from transformers import WhisperForConditionalGeneration, WhisperProcessor\n\n    dtype = getattr(torch, dtype_name, None) or torch.float32\n    processor = WhisperProcessor.from_pretrained(snapshot_path, local_files_only = True)","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/inference/stt_transformers_worker.py#L79-L115","documentation":"Raised in the STT worker child process by _raise_if_cancelled when the shared cancel event is set during model loading. The parent sets the event to abort a long Whisper load so GPU/memory can be handed to training. SttLoadCancelledError signals deliberate preemption, not corruption or a broken model.","triggerScenarios":"A load_whisper(...) call in the worker child with a cancel_event that the parent process sets while weights are still being read from the Hub cache and moved to the device.","commonSituations":"A scheduler that preempts STT model loads when a training job starts; low-VRAM hosts where loads are slow and often interrupted; rapid switching between dictation and training features.","solutions":["Catch SttLoadCancelledError in the parent and retry the load when the GPU is free (optionally via the existing load API with a fresh event).","If loads are cancelled repeatedly, schedule STT loads outside training windows or on a quieter device.","Do not reuse the cancelled event; create a new one for the retry."],"exampleFix":"# before\nworker.load(model_id, cancel_event=evt)  # raises mid-load when training starts\n\n# after\ntry:\n    worker.load(model_id, cancel_event=evt)\nexcept SttLoadCancelledError:\n    evt = threading.Event()  # fresh event\n    worker.load(model_id, cancel_event=evt)  # retry after training frees the GPU","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"from core.inference.stt_sidecar import SttLoadCancelledError\ntry:\n    stt.load(model_id, cancel_event=evt)\nexcept SttLoadCancelledError:\n    evt = threading.Event()\n    schedule_retry_when_gpu_free(lambda: stt.load(model_id, cancel_event=evt))","preventionTips":["Do not start STT loads while a training job holds or is about to grab the GPU.","Pass a per-load cancel event so old events cannot abort fresh loads.","Track whether training is active in your scheduler before requesting a load."],"tags":["stt","cancellation","multiprocessing","gpu"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}