{"record":{"id":"8604daa5f9f85fb0","repo":"zylon-ai/private-gpt","slug":"item-transformation-failed-e-s","errorCode":null,"errorMessage":"Item transformation failed: {e!s}","messagePattern":"Item transformation failed: (.+?)","errorType":"exception","errorClass":"AsyncIteratorError","httpStatus":null,"severity":"error","filePath":"private_gpt/utils/async_utils.py","lineNumber":87,"sourceCode":"                        f\"Iterator next() operation failed: {e!s}\"\n                    ) from e\n\n            if not chunk:\n                break\n\n            # Process the chunk\n            for item in chunk:\n                try:\n                    if transform_fn:\n                        # Run transform in executor if it's CPU-intensive\n                        result = await loop.run_in_executor(\n                            internal_executor, transform_fn, item\n                        )\n                        yield result\n                    else:\n                        yield item\n                except Exception as e:\n                    raise AsyncIteratorError(\n                        f\"Item transformation failed: {e!s}\"\n                    ) from e\n\n    except asyncio.CancelledError:\n        raise\n    except Exception as e:\n        raise AsyncIteratorError(f\"Async iterator conversion failed: {e!s}\") from e\n    finally:\n        if not executor:\n            internal_executor.shutdown(wait=False)\n","sourceCodeStart":69,"sourceCodeEnd":98,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/utils/async_utils.py#L69-L98","documentation":"Wrapped as AsyncIteratorError by to_async_iterator when the user-supplied transform_fn raises while processing an item. transform_fn is executed in the thread pool through loop.run_in_executor; any exception it throws (TypeError, KeyError, unexpected input shape) is caught, re-raised as AsyncIteratorError('Item transformation failed: ...'), and the async generator terminates - no later items are produced.","triggerScenarios":"Passing transform_fn=lambda d: d['text'] when some items lack the 'text' key; a transform that deserializes JSON and hits malformed payload; type mismatches between what the iterator yields and what transform_fn expects; a transform with side effects (HTTP call) that fails transiently.","commonSituations":"Document-processing pipelines (chunkers, embedders, parsers) where one malformed document kills the whole stream; upgrading a dependency that changes item schema so the transform no longer matches; LLM response parsing where the model occasionally returns an unexpected structure.","solutions":["Make transform_fn total: wrap its body in try/except and return a fallback (e.g. None or an error-record) for bad items, then filter downstream","Validate/normalize items before passing them to to_async_iterator","Add logging inside transform_fn including the offending item so failures are diagnosable","If failures are transient (network), add retry with backoff inside the transform instead of letting it raise"],"exampleFix":"# before\nait = to_async_iterator(iter(docs), transform_fn=lambda d: d['text'])\n\n# after\ndef get_text(d):\n    try:\n        return d['text']\n    except KeyError:\n        logger.warning(f\"item missing text: {d!r}\")\n        return None\nait = to_async_iterator(iter(docs), transform_fn=get_text)","handlingStrategy":"try-catch","validationCode":"def safe_transform(fn, fallback=None):\n    def wrapper(item):\n        try:\n            return fn(item)\n        except Exception as e:\n            logger.warning(\"transform failed for %r: %s\", item, e)\n            return fallback\n    return wrapper\n\n# pass wrapper instead of fn to to_async_iterator","typeGuard":"null","tryCatchPattern":"try:\n    async for out in to_async_iterator(it, transform_fn=fn):\n        ...\nexcept AsyncIteratorError as e:\n    if \"Item transformation\" in str(e):\n        logger.error(\"bad item: %s\", e.__cause__)\n        # decide: skip item (needs defensive fn) or abort stream","preventionTips":["Design transform_fn to be total: return a sentinel for bad inputs instead of raising","Unit-test transforms against malformed sample items (missing keys, None, wrong types)","Filter/normalize items before passing them into to_async_iterator","Keep transforms pure and cheap; put retries inside them for transient I/O"],"tags":["async","iterator","transformation","error-wrapping"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}