{"record":{"id":"31b29155fbd9f460","repo":"unslothai/unsloth","slug":"failed-to-check-dataset-format-scrubbed","errorCode":null,"errorMessage":"Failed to check dataset format: {scrubbed}","messagePattern":"Failed to check dataset format: (.+?)","errorType":"http","errorClass":"HTTPException","httpStatus":500,"severity":"error","filePath":"studio/backend/hub/services/datasets/formatting.py","lineNumber":531,"sourceCode":"    except Exception as e:\n        scrubbed = download_registry.scrub_secrets(str(e), hf_token = hf_token)\n        # Missing/gated/bad-token and malformed names are client errors, not 500s.\n        status = hf_error_status(e)\n        if (\n            status is None\n            and isinstance(e, OSError)\n            and getattr(e, \"errno\", None) == errno.ENAMETOOLONG\n        ):\n            status, scrubbed = 400, \"Invalid dataset name\"\n        elif status is None and isinstance(e, FileNotFoundError):\n            # datasets raises DatasetNotFoundError (FileNotFoundError) for missing/gated.\n            status = 404\n        elif status is None and isinstance(e, ValueError):\n            status = 400\n        if status is not None:\n            raise HTTPException(status_code = status, detail = scrubbed)\n        logger.error(\"Error checking dataset format: %s\", scrubbed)\n        raise HTTPException(\n            status_code = 500,\n            detail = \"Failed to check dataset format: \" + scrubbed,\n        )\n\n\ndef ai_assist_mapping_response(\n    request: AiAssistMappingRequest, hf_token: Optional[str] = None\n) -> AiAssistMappingResponse:\n    \"\"\"\n    Run the LLM-assisted dataset conversion advisor (user-triggered).\n\n    Multi-pass analysis with a 7B helper model: classify dataset type, generate\n    a conversion strategy, then validate it. Falls back to simple column\n    classification if the advisor fails.\n    \"\"\"\n    try:\n        from hub.utils.llm_assist import llm_conversion_advisor\n","sourceCodeStart":513,"sourceCodeEnd":549,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/hub/services/datasets/formatting.py#L513-L549","documentation":"HTTP 500 raised at the end of the dataset format-check handler when the caught exception matched none of the mapped client-error branches (ENAMETOOLONG→400 invalid name, FileNotFoundError/DatasetNotFound→404, ValueError→400, hf_error_status mappings). The original message is scrubbed of secrets (`scrub_secrets`) and appended; the raw error is also logged at error level with the scrubbed text.","triggerScenarios":"Any unexpected exception during format inspection: a corrupt parquet file raising pyarrow errors, an unexpected HTTP status from the hub, an arrow memory error, or a library incompatibility — anything that is not NotFound/ValueError/OSError-ENAMETOOLONG.","commonSituations":"Corrupt or truncated downloaded shards; pyarrow/datasets version mismatch after an upgrade; huge datasets hitting OOM during schema inference; transient hub 5xx that hf_error_status does not classify.","solutions":["Read the scrubbed message in the 500 detail and the full traceback the server logged (`Error checking dataset format: ...`) — the detail is only the exception string.","If it names a corrupt file, delete the dataset cache and re-download.","If it names a version/library issue, pin matching pyarrow/datasets versions and restart the backend.","For transient hub errors, retry after a short wait."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    fmt = check_dataset_format(client, req)\nexcept HTTPStatusError as e:\n    if e.response.status_code >= 500:\n        log_server_trace_id(); notify_user_retryable()  # scrubbed detail + server log has traceback\n    else:\n        handle_client_error(e)  # 400/404 mappings are actionable","preventionTips":["Treat 4xx from this endpoint as dataset problems and 5xx as environment problems.","Keep pyarrow/datasets versions aligned with the backend's requirements.","Re-download caches when corrupt-shard errors appear in the scrubbed detail."],"tags":["http-500","huggingface","format-check","unexpected-exception","parquet"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}