{"record":{"id":"a2abe98c17fc21ea","repo":"karpathy/nanochat","slug":"unknown-part-type-part-type","errorCode":null,"errorMessage":"Unknown part type: {part['type']}","messagePattern":"Unknown part type: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"nanochat/tokenizer.py","lineNumber":216,"sourceCode":"                elif isinstance(content, list):\n                    for part in content:\n                        value_ids = self.encode(part[\"text\"])\n                        if part[\"type\"] == \"text\":\n                            # string part => simply add the tokens\n                            add_tokens(value_ids, 1)\n                        elif part[\"type\"] == \"python\":\n                            # python tool call => add the tokens inside <|python_start|> and <|python_end|>\n                            add_tokens(python_start, 1)\n                            add_tokens(value_ids, 1)\n                            add_tokens(python_end, 1)\n                        elif part[\"type\"] == \"python_output\":\n                            # python output => add the tokens inside <|output_start|> and <|output_end|>\n                            # none of these tokens are supervised because the tokens come from Python at test time\n                            add_tokens(output_start, 0)\n                            add_tokens(value_ids, 0)\n                            add_tokens(output_end, 0)\n                        else:\n                            raise ValueError(f\"Unknown part type: {part['type']}\")\n                else:\n                    raise ValueError(f\"Unknown content type: {type(content)}\")\n                add_tokens(assistant_end, 1)\n\n        # truncate to max_tokens tokens MAX (helps prevent OOMs)\n        ids = ids[:max_tokens]\n        mask = mask[:max_tokens]\n        return ids, mask\n\n    def visualize_tokenization(self, ids, mask, with_token_id=False):\n        \"\"\"Small helper function useful in debugging: visualize the tokenization of render_conversation\"\"\"\n        RED = '\\033[91m'\n        GREEN = '\\033[92m'\n        RESET = '\\033[0m'\n        GRAY = '\\033[90m'\n        tokens = []\n        for i, (token_id, mask_val) in enumerate(zip(ids, mask)):\n            token_str = self.decode([token_id])","sourceCodeStart":198,"sourceCodeEnd":234,"githubUrl":"https://github.com/karpathy/nanochat/blob/92d63d4e8bb4df75c3b71618f31ddde2378b2bcd/nanochat/tokenizer.py#L198-L234","documentation":"In `render_conversation`, an assistant message's content may be a list of typed parts. The supported part types are 'text', 'python' (a supervised tool call wrapped in <|python_start|>/<|python_end|>), and 'python_output' (unsupervised tool output wrapped in <|output_start|>/<|output_end|>). Any other value in part['type'] raises ValueError.","triggerScenarios":"Feeding a chat SFT dataset where assistant parts use other type labels — e.g. OpenAI-style 'tool_calls'/'function_call', 'tool_response', 'code', or a typo like 'pythonoutput' — into tokenizer.render_conversation.","commonSituations":"Converting external chat datasets (ShareGPT/OpenAI format) into nanochat's conversation format without mapping tool-call fields; hand-writing synthetic conversations with invented part types; dataset schema drift after an upstream update.","solutions":["Map every assistant part to type 'text', 'python', or 'python_output' when preparing data.","Convert OpenAI tool_calls entries: the call arguments become a 'python' part, the tool result becomes a 'python_output' part, plain prose becomes 'text'.","Write a small pre-flight check over your dataset (see validation code) before training."],"exampleFix":"# before\n{\"role\": \"assistant\", \"content\": [{\"type\": \"tool_call\", \"text\": \"2+2\"}]}\n\n# after\n{\"role\": \"assistant\", \"content\": [{\"type\": \"python\", \"text\": \"2+2\"}]}","handlingStrategy":"validation","validationCode":"VALID_PART_TYPES = {'text', 'python', 'python_output'}\nfor msg in conversation[\"messages\"]:\n    if isinstance(msg.get(\"content\"), list):\n        for part in msg[\"content\"]:\n            assert part.get(\"type\") in VALID_PART_TYPES, f\"bad part type {part.get('type')!r}\"","typeGuard":"def is_valid_part(part) -> bool:\n    return isinstance(part, dict) and part.get('type') in {'text', 'python', 'python_output'} and isinstance(part.get('text'), str)","tryCatchPattern":null,"preventionTips":["Map external chat formats (OpenAI tool_calls, ShareGPT) to nanochat part types at data-prep time.","Validate a sample of conversations with render_conversation during dataset builds, before training.","Use visualize_tokenization on a few rendered docs to eyeball the mapping."],"tags":["nanochat","tokenizer","sft-data","tool-use","validation"],"backgroundTag":null,"analyzedSha":"92d63d4e8bb4df75c3b71618f31ddde2378b2bcd","analyzedAt":"2026-08-15T03:11:54.371Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}