{"record":{"id":"b81bce02b3476567","repo":"pathwaycom/pathway","slug":"unsupported-conversion-from-pw-json-to-typehints","errorCode":null,"errorMessage":"Unsupported conversion from pw.Json to {typehints[name]}","messagePattern":"Unsupported conversion from pw\\.Json to (.+?)","errorType":"validation","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"python/pathway/stdlib/utils/col.py","lineNumber":176,"sourceCode":"                result = _optional(\n                    col,\n                    lambda col: pw.unwrap(col.as_str()).dt.strptime(\n                        \"%Y-%m-%dT%H:%M:%S.%f\"\n                    ),\n                )\n            case dt.DATE_TIME_UTC:\n                result = _optional(\n                    col,\n                    lambda col: pw.unwrap(col.as_str()).dt.strptime(\n                        \"%Y-%m-%dT%H:%M:%S.%f%z\"\n                    ),\n                )\n            case dt.DURATION:\n                result = _optional(\n                    col, lambda col: pw.unwrap(col.as_int()).dt.to_duration(\"ns\")\n                )\n            case _:\n                raise TypeError(\n                    f\"Unsupported conversion from pw.Json to {typehints[name]}\"\n                )\n\n        return result if is_optional else pw.unwrap(result)\n\n    colrefs = [pw.this[column_name] for column_name in schema.column_names()]\n    kw = {\n        colref.name: _convert_from_json(colref.name, column.get(colref.name))\n        for colref in colrefs\n    }\n    result = column.table.select(**kw).update_types(**schema)\n    return result\n\n\n# TODO: generalize to apply on groupby: https://github.com/navalgo/IoT-Pathway/issues/1919\n@check_arg_types\n@trace_user_frame\ndef multiapply_all_rows(","sourceCodeStart":158,"sourceCodeEnd":194,"githubUrl":"https://github.com/pathwaycom/pathway/blob/fa2f74a4649b7c5908690cf60137263d8d80de5f/python/pathway/stdlib/utils/col.py#L158-L194","documentation":"The JSON-to-schema conversion helper in pw.utils.col builds column expressions that extract typed values from a pw.Json column according to a target schema. Only a fixed set of target dtypes has an extraction recipe (e.g. strings, ints, floats, bools, DATE_TIME_UTC via strptime, DURATION via int nanoseconds); the final `case _` raises TypeError naming the unsupported target type.","triggerScenarios":"Calling the json-unpacking API (pw.utils.col based, e.g. unpack/parse with schema=) where the schema declares bytes, a Pointer, an array/map dtype, or another exotic type; targeting DATE_TIME_NAIVE patterns that do not match the supported strptime format when no case exists for that dtype.","commonSituations":"Feeding ORM/dataclass schemas containing bytes or UUID-like fields into a JSON unpacker; schema-first design where the schema was generated from a DB model rather than restricted to JSON-representable types; API responses with nested objects mapped to columns not covered by the converter.","solutions":["Restrict the schema passed to the conversion to the supported dtypes (str/int/float/bool/datetime/duration) and unpack exotic fields manually","For unsupported types, extract via column.get('field') and convert with .apply / .dt helpers instead of schema-driven conversion","Store genuinely non-primitive fields as pw.Json or str in the target schema and decode downstream"],"exampleFix":"# before\nclass S(pw.Schema):\n    id: bytes  # unsupported target\nout = convert(t.raw_json, schema=S)\n\n# after\nclass S(pw.Schema):\n    id: str\nout = convert(t.raw_json, schema=S)","handlingStrategy":"type-guard","validationCode":"from pathway import dt\nSUPPORTED = {dt.STRING, dt.INT, dt.FLOAT, dt.BOOL, dt.DATE_TIME_UTC, dt.DATE_TIME_NAIVE, dt.DURATION}\nbad = [n for n, t in schema.typehints().items() if dt.eval_type(t) not in SUPPORTED]\nassert not bad, f'unsupported target dtypes for json conversion: {bad}'","typeGuard":"def schema_is_json_convertible(schema: type) -> bool:\n    from pathway import dt\n    SUPPORTED = {dt.STRING, dt.INT, dt.FLOAT, dt.BOOL, dt.DATE_TIME_UTC, dt.DATE_TIME_NAIVE, dt.DURATION}\n    return all(dt.eval_type(t) in SUPPORTED for t in schema.typehints().values())","tryCatchPattern":"try:\n    result = convert(json_col, schema=S)\nexcept TypeError as e:\n    if 'Unsupported conversion from pw.Json' in str(e):\n        raise TypeError(f'schema {S} has a non-JSON-representable column') from e\n    raise","preventionTips":["Keep JSON-extraction schemas limited to primitives (str/int/float/bool/datetime/duration)","Move bytes/array/nested fields out of the schema and extract them with column.get(...) + apply","Add a unit test asserting the schema passes the supported-dtype check"],"tags":["pathway","json","type-conversion","schema"],"backgroundTag":null,"analyzedSha":"fa2f74a4649b7c5908690cf60137263d8d80de5f","analyzedAt":"2026-08-15T01:48:17.006Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}