{"record":{"id":"24e2511133fae898","repo":"run-llama/liteparse","slug":"parse-of-source-exceeded-timeout-s-the-worker","errorCode":null,"errorMessage":"parse of {source} exceeded {timeout}s; the worker process was killed","messagePattern":"parse of (.+?) exceeded (.+?)s; the worker process was killed","errorType":"exception","errorClass":"ParseTimeoutError","httpStatus":null,"severity":"error","filePath":"packages/python/liteparse/_pool.py","lineNumber":245,"sourceCode":"            if worker in self._workers:\n                self._workers.remove(worker)\n\n    def parse(self, payload: Union[str, bytes], source: str) -> Any:\n        \"\"\"Run one parse on an idle worker.\n\n        Blocks until a worker is free; ``parse_timeout`` bounds the parse\n        itself, not the wait for a free worker.\n        \"\"\"\n        if self._closed:\n            raise ParseError(\"parser pool is closed\")\n        worker = self._idle.get()\n        replace = False\n        try:\n            status, data = worker.request(payload, self._timeout)\n        except _WorkerTimeout:\n            replace = True\n            timeout = self._timeout\n            raise ParseTimeoutError(\n                f\"parse of {source} exceeded {timeout}s; \"\n                \"the worker process was killed\",\n                source=source,\n                timeout=timeout,\n            ) from None\n        except _WorkerCrashed as e:\n            replace = True\n            raise ParseError(\n                f\"liteparse worker process died while parsing {source}: {e}\"\n            ) from None\n        except _WorkerInitFailed as e:\n            replace = True\n            raise ParseError(f\"liteparse worker failed to initialize: {e}\") from None\n        finally:\n            if replace:\n                self._retire_worker(worker)\n                if not self._closed:\n                    self._spawn_worker()","sourceCodeStart":227,"sourceCodeEnd":263,"githubUrl":"https://github.com/run-llama/liteparse/blob/22d2dd8cd7f7b9320102b57ddaf0e663ff7d15a8/packages/python/liteparse/_pool.py#L227-L263","documentation":"When a parse exceeds the pool's parse_timeout, the worker subprocess is killed and parse raises ParseTimeoutError carrying the source name and the timeout value. The pool then transparently replaces the killed worker so subsequent parses still work. This is a deliberate hard-kill design: a hung parse (e.g. inside PDFium) cannot be interrupted in-process, so the whole worker is terminated.","triggerScenarios":"Calling WorkerPool.parse on a document that takes longer than parse_timeout seconds — huge or pathological PDFs, PDFs with pathological content streams that hang PDFium, or a parse_timeout set too low for your document sizes (or left as a small default).","commonSituations":"Scanning very large scanned PDFs with OCR-heavy pages; a corrupt/malformed PDF causing PDFium to loop; setting parse_timeout=5s while real documents take 30s; batch jobs where one bad file kills its worker repeatedly.","solutions":["Raise parse_timeout to a value comfortably above your worst-case document parse time (or pass None to disable)","Pre-screen inputs: skip or route oversized/complex PDFs (page count, file size) to a longer-timeout pool","Catch ParseTimeoutError per document and continue the batch — the pool already replaced the dead worker","Try parsing the offending file with the native CLI to determine whether it hangs deterministically; report pathological files upstream","Run a separate pool with a larger timeout for heavy documents instead of slowing every request"],"exampleFix":"# before\npool = WorkerPool(config, pool_size=4, parse_timeout=10)\nresult = pool.parse(big_pdf_bytes, 'huge.pdf')  # ParseTimeoutError\n# after\npool = WorkerPool(config, pool_size=4, parse_timeout=120)\ntry:\n    result = pool.parse(big_pdf_bytes, 'huge.pdf')\nexcept ParseTimeoutError as e:\n    log.warning('skipped %s: exceeded %ss', e.source, e.timeout)\n    result = None","handlingStrategy":"try-catch","validationCode":"MAX_PAGES = 2000\nif os.path.getsize(path) > 500 * 1024 * 1024:\n    raise ValueError(f'{path} too large for default timeout pool')","typeGuard":"from liteparse import ParseTimeoutError\n\ndef is_timeout_error(e: Exception) -> bool:\n    return isinstance(e, ParseTimeoutError)","tryCatchPattern":"try:\n    result = pool.parse(data, source)\nexcept ParseTimeoutError as e:\n    log.warning('parse timeout for %s after %ss', e.source, e.timeout)\n    result = None  # or retry on a long-timeout pool","preventionTips":["Set parse_timeout from measured worst-case parse times, not guesses","Route very large or scanned documents to a dedicated pool with a larger timeout","Catch ParseTimeoutError per item in batch jobs so one bad file doesn't abort the run","Provision headroom on CPU-constrained machines; slow hosts need larger timeouts","Report deterministically hanging PDFs to the liteparse maintainers"],"tags":["python","timeout","subprocess","pdf","concurrency"],"backgroundTag":"request-timeout","analyzedSha":"22d2dd8cd7f7b9320102b57ddaf0e663ff7d15a8","analyzedAt":"2026-09-08T06:09:49.009Z","contentChangedAt":"2026-09-08T06:09:49.009Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}