{"record":{"id":"1b16405bc6592485","repo":"zylon-ai/private-gpt","slug":"document-extraction-unsuccessful-for-file-name","errorCode":null,"errorMessage":"Document extraction unsuccessful for '{file_name}': unmapped-glyph ratio exceeded threshold ({self.config.failure_threshold}).","messagePattern":"Document extraction unsuccessful for '(.+?)': unmapped-glyph ratio exceeded threshold \\((.+?)\\)\\.","errorType":"exception","errorClass":"ExtractionUnsuccessfulError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/readers/docling/docling_api_reader.py","lineNumber":183,"sourceCode":"        try:\n            conversion_result = await self.client.convert_from_bytes(\n                file_name, file_bytes, to_formats=[\"md\"], pages=pages, **load_kwargs\n            )\n        except Exception as e:\n            raise ValueError(f\"Document conversion failed: {e}\") from e\n\n        if conversion_result.status not in [\"success\", \"partial_success\"]:\n            raise ValueError(\n                f\"Document conversion failed with status: {conversion_result.status}. \"\n                f\"Errors: {conversion_result.errors}\"\n            )\n\n        contents = self._get_content(conversion_result)\n        valid_contents = [content for content in contents if content]\n        if not valid_contents:\n            raise ValueError(\"No valid document content found after conversion\")\n        if self._is_extraction_unsuccessful(valid_contents):\n            raise ExtractionUnsuccessfulError(\n                f\"Document extraction unsuccessful for '{file_name}': unmapped-glyph \"\n                f\"ratio exceeded threshold ({self.config.failure_threshold}).\"\n            )\n\n        docs = [\n            self._page_to_doc(\n                content=content,\n                index=idx,\n                include_page_metadata=len(valid_contents) > 1,\n                extra_info=extra_info,\n            )\n            for idx, content in enumerate(valid_contents)\n        ]\n\n        if debug_mode:\n            logger.info(f\"Loaded document from {file_name}\")\n            logger.info(f\"Document has {len(docs)} pages\")\n            if conversion_result.timings:","sourceCodeStart":165,"sourceCodeEnd":201,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/readers/docling/docling_api_reader.py#L165-L201","documentation":"Raised as ExtractionUnsuccessfulError by DoclingApiReader.lazy_load_data when _is_extraction_unsuccessful(valid_contents) detects that the ratio of unmapped glyphs exceeds the configured failure_threshold (from the reader's transformation.docling config). It guards against garbage ingestion: PDFs whose fonts fail to map to real text often convert 'successfully' but produce replacement-character junk, which would poison the vector index.","triggerScenarios":"Ingesting PDFs with broken/non-embedded fonts (common with old scans, some LaTeX/Windows font subsets, and CAD exports) where conversion succeeds but the text contains a high proportion of unmapped-glyph placeholder characters, pushing the ratio over config.failure_threshold.","commonSituations":"Batch-ingesting third-party/scanned archives; documents from plotter/CAD tools; threshold too strict after upgrading private-gpt (default sensitivity change); the same documents previously ingested before the guard was added.","solutions":["Inspect the extracted text for the file — if it is replacement-character junk, re-OCR the source (enable force_full_page_ocr / use_ocr) or obtain a better source PDF.","If the document's text is acceptable and the guard is too aggressive for your corpus, raise failure_threshold in the docling transformation settings.","Handle ExtractionUnsuccessfulError specifically in the ingestion loop and route the file to manual review/quarantine rather than failing the batch.","For PDFs you control, re-export with embedded fonts to fix the root cause."],"exampleFix":"# ingestion loop — before\nnodes = await reader.lazy_load_data(file_info)\n\n# after\nfrom private_gpt.components.readers.docling.docling_api_reader import ExtractionUnsuccessfulError\ntry:\n    nodes = await reader.lazy_load_data(file_info)\nexcept ExtractionUnsuccessfulError:\n    quarantine(file_info)  # manual review path\n    return []","handlingStrategy":"try-catch","validationCode":"def unmapped_glyph_ratio(text: str) -> float:\n    if not text:\n        return 0.0\n    bad = sum(text.count(c) for c in \"\\ufffd\\u25a1\")\n    return bad / max(len(text), 1)\n\n# pre-check before accepting a converted doc into the index\nratio = unmapped_glyph_ratio(extracted_text)\nif ratio > config.failure_threshold:\n    route_to_manual_review(file_name)","typeGuard":null,"tryCatchPattern":"from private_gpt.components.readers.docling.docling_api_reader import ExtractionUnsuccessfulError\n\ntry:\n    nodes = [n async for n in reader.lazy_load_data(file_info)]\nexcept ExtractionUnsuccessfulError:\n    quarantine(file_info, reason=\"unmapped-glyph ratio over threshold\")\n    return []  # keep the batch alive; this file needs a better source or OCR","preventionTips":["Catch ExtractionUnsuccessfulError separately from ValueError at the ingestion loop.","Tune transformation.docling failure_threshold to your corpus quality instead of disabling the guard.","Prefer fixing the source (re-export with embedded fonts, or full-page OCR) over lowering thresholds indiscriminately."],"tags":["docling","data-quality","fonts","ingestion"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}