{"record":{"id":"6907d935b3466155","repo":"mudler/LocalAI","slug":"dataset-source-is-required-path-to-a-preprocessed","errorCode":null,"errorMessage":"dataset_source is required (path to a preprocessed dataset)","messagePattern":"dataset_source is required \\(path to a preprocessed dataset\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"backend/python/liquid-audio/backend.py","lineNumber":646,"sourceCode":"            job.completed = True\n            print(f\"Training failed: {exc}\", file=sys.stderr)\n            print(traceback.format_exc(), file=sys.stderr)\n            job.progress_queue.put(backend_pb2.FineTuneProgressUpdate(\n                job_id=job.job_id, status=\"failed\", message=str(exc),\n            ))\n        finally:\n            job.progress_queue.put(None)\n\n    def _do_train(self, request, job):\n        from liquid_audio import LFM2AudioModel  # noqa: F401  (sanity import)\n        from liquid_audio.data.dataloader import LFM2DataLoader\n        from liquid_audio.trainer import Trainer\n\n        model_id = request.model or self.model_id or \"LiquidAI/LFM2.5-Audio-1.5B\"\n\n        dataset_path = request.dataset_source\n        if not dataset_path:\n            raise ValueError(\"dataset_source is required (path to a preprocessed dataset)\")\n\n        extras = dict(request.extra_options) if request.extra_options else {}\n        val_path = extras.get(\"val_dataset\")\n\n        # Map FineTuneRequest hyperparameters to liquid_audio.Trainer constructor args\n        lr = request.learning_rate or 3e-5\n        max_steps = request.max_steps or 1000\n        warmup_steps = request.warmup_steps or min(100, max_steps // 10)\n        batch_size = request.batch_size or 16\n        save_interval = request.save_steps or max(1, max_steps // 4)\n\n        output_dir = request.output_dir or os.path.join(\n            os.environ.get(\"LIQUID_AUDIO_OUTPUT_DIR\", \"/tmp\"),\n            f\"liquid-audio-{job.job_id}\",\n        )\n        os.makedirs(output_dir, exist_ok=True)\n\n        job.progress_queue.put(backend_pb2.FineTuneProgressUpdate(","sourceCodeStart":628,"sourceCodeEnd":664,"githubUrl":"https://github.com/mudler/LocalAI/blob/44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26/backend/python/liquid-audio/backend.py#L628-L664","documentation":"ValueError from the liquid-audio backend's training path (_do_train): the FineTune gRPC request must carry dataset_source pointing to an already-preprocessed dataset, because the liquid_audio Trainer's LFM2DataLoader consumes preprocessed data rather than raw audio/text. An empty or missing dataset_source aborts the fine-tune job before any training starts.","triggerScenarios":"Sending a FineTune request with dataset_source unset/empty; assuming the backend preprocesses raw datasets itself; passing the path under a different request field (e.g. only in extra_options).","commonSituations":"User points at a raw folder of wav/json files expecting preprocessing; the preprocessed output directory name was mistyped or the preprocessing job wrote elsewhere; dataset_source passed only via extra_options instead of the dedicated field.","solutions":["Run the preprocessing step first and pass its output directory as request.dataset_source","Verify the path exists and contains the preprocessed artifacts the LFM2DataLoader expects before submitting the job","If you meant to validate only, skip FineTune — this field is mandatory for any training run"],"exampleFix":"# before\nrequest = backend_pb2.FineTuneRequest(base_model=model_id)  # dataset_source omitted\n\n# after\nrequest = backend_pb2.FineTuneRequest(\n    base_model=model_id,\n    dataset_source=\"/data/lfm2_audio_preprocessed/train\",\n)","handlingStrategy":"validation","validationCode":"import os\n\ndef validate_finetune_request(req) -> None:\n    ds = getattr(req, \"dataset_source\", \"\")\n    if not ds:\n        raise ValueError(\"dataset_source is required\")\n    if not os.path.isdir(ds):\n        raise FileNotFoundError(f\"preprocessed dataset not found: {ds}\")","typeGuard":null,"tryCatchPattern":"try:\n    stub.FineTune(request)\nexcept grpc.RpcError as e:\n    if \"dataset_source is required\" in (e.details() or \"\"):\n        # surface a clear UI error about the missing preprocessing step\n        raise UserError(\"Run dataset preprocessing first\") from e\n    raise","preventionTips":["Make preprocessing a mandatory pipeline stage that outputs the path you then pass verbatim","Assert the preprocessed directory is non-empty before submitting the training job"],"tags":["python","liquid-audio","fine-tuning","grpc","validation"],"backgroundTag":null,"analyzedSha":"44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26","analyzedAt":"2026-08-15T10:13:50.291Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}