{"record":{"id":"6c7a037a170448dd","repo":"unslothai/unsloth","slug":"fail-rate-0-of-images-failed-to-download","errorCode":null,"errorMessage":"⚠️ {fail_rate:.0%} of images failed to download ({failed_count}/{total}). This dataset has too many broken or unreachable image URLs. Consider using a dataset with embedded images instead.","messagePattern":"⚠️ (.+?) of images failed to download \\((.+?)/(.+?)\\)\\. This dataset has too many broken or unreachable image URLs\\. Consider using a dataset with embedded images instead\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"studio/backend/utils/datasets/format_conversion.py","lineNumber":673,"sourceCode":"            ]\n            friendly = None\n            try:\n                from .llm_assist import llm_generate_dataset_warning\n                friendly = llm_generate_dataset_warning(\n                    issues,\n                    dataset_name = dataset_name,\n                    modality = \"vision\",\n                    column_names = [image_column, text_column],\n                )\n            except Exception:\n                pass\n            msg = friendly or (\n                f\"⚠️ {fail_rate:.0%} of images failed to download ({failed_count}/{total}). \"\n                \"This dataset has too many broken or unreachable image URLs. \"\n                \"Consider using a dataset with embedded images instead.\"\n            )\n            _notify(msg)\n            raise ValueError(msg)\n\n    if len(converted_list) == 0:\n        issues = [\n            f\"All {total} samples failed during VLM conversion — no usable images found\",\n            f\"Image column '{image_column}' may contain URLs that are no longer accessible, \"\n            \"or local file paths that don't exist\",\n        ]\n        friendly = None\n        try:\n            from .llm_assist import llm_generate_dataset_warning\n            friendly = llm_generate_dataset_warning(\n                issues,\n                dataset_name = dataset_name,\n                modality = \"vision\",\n                column_names = [image_column, text_column],\n            )\n        except Exception:\n            pass","sourceCodeStart":655,"sourceCodeEnd":691,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/utils/datasets/format_conversion.py#L655-L691","documentation":"Raised after the bulk image download phase of VLM dataset conversion when the overall failure rate across all samples exceeds the allowed threshold. Unlike the probe-stage error, this fires once the full (or remainder) download has run and too many URLs failed. It aborts the conversion with a ValueError containing either an LLM-friendly warning or the default formatted message via _notify().","triggerScenarios":"Running VLM conversion over a dataset whose image column contains URLs, where the probe batch passed but the aggregate failed_count/total ratio crosses the rejection threshold (e.g. links that die mid-download, CDN throttling after N requests, mixed good/dead URL batches).","commonSituations":"Large datasets scraped long ago with progressive link rot; image hosts that start returning 403/429 after bulk fetching; flaky networks where sustained throughput drops cause mass timeouts; datasets mixing accessible and expired signed URLs.","solutions":["Re-run conversion with fewer download workers to avoid CDN rate limiting (worker count is shown in the info message before download).","Check network egress / proxy settings; a proxy that drops sustained parallel connections produces exactly this pattern.","Filter the dataset first: probe URLs with a small HEAD-request script, keep only reachable rows, then convert.","Switch to a dataset with embedded image bytes (HF datasets with an image feature) to avoid network dependency entirely.","If links are signed/expired, re-generate the URL column from the source dataset revision."],"exampleFix":"# before\nresult = convert_dataset(images_url_ds, num_workers=32)  # mass 429s -> ValueError\n\n# after\nresult = convert_dataset(images_url_ds, num_workers=4)  # stays under rate limit","handlingStrategy":"retry","validationCode":"def safe_convert(ds, image_column, worker_steps=(16, 8, 2)):\n    last = None\n    for workers in worker_steps:\n        try:\n            return convert_dataset(ds, image_column=image_column, num_workers=workers)\n        except ValueError as e:\n            if \"failed to download\" not in str(e):\n                raise\n            last = e\n    raise last","typeGuard":null,"tryCatchPattern":"try:\n    result = convert_dataset(ds, image_column=\"image_url\")\nexcept ValueError as e:\n    if \"too many broken\" in str(e):\n        log.warning(\"Image host throttling suspected; retrying with 2 workers\")\n        result = convert_dataset(ds, image_column=\"image_url\", num_workers=2)\n    else:\n        raise","preventionTips":["Start bulk image downloads with conservative worker counts to avoid CDN 429s.","Monitor the 'X% broken URLs will be skipped' notice mid-run and abort early if it climbs.","Cache downloaded images locally keyed by URL hash so reruns skip network.","Keep converted snapshots of URL datasets so a one-time success is reusable."],"tags":["dataset","vlm","network","rate-limit","image-download"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}