{"record":{"id":"0e5c7f8b5e5ab2e1","repo":"hankcs/HanLP","slug":"unknown-data-arrangement","errorCode":null,"errorMessage":"Unknown data arrangement","messagePattern":"Unknown data arrangement","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"hanlp/transform/conll_tf.py","lineNumber":698,"sourceCode":"        self.form_vocab.add(ROOT)  # make root the 2ed elements while 0th is pad, 1st is unk\n        if self.use_pos:\n            self.cpos_vocab = VocabTF(pad_token=None, unk_token=None)\n        self.rel_vocab = VocabTF(pad_token=None, unk_token=None)\n        num_samples = 0\n        counter = Counter()\n        for sent in self.file_to_samples(trn_path, gold=True):\n            num_samples += 1\n            for idx, cell in enumerate(sent):\n                if len(cell) == 4:\n                    form, cpos, head, deprel = cell\n                elif len(cell) == 3:\n                    if self.use_pos:\n                        form, cpos = cell[0]\n                    else:\n                        form = cell[0]\n                    head, deprel = cell[1:]\n                else:\n                    raise ValueError('Unknown data arrangement')\n                if idx == 0:\n                    root = form\n                else:\n                    counter[form] += 1\n                if self.use_pos:\n                    self.cpos_vocab.add(cpos)\n                self.rel_vocab.update(deprel)\n\n        for token in [token for token, freq in counter.items() if freq >= self.config.min_freq]:\n            self.form_vocab.add(token)\n        return num_samples\n\n    def inputs_to_samples(self, inputs, gold=False):\n        use_pos = self.use_pos\n        for sent in inputs:\n            sample = []\n            for i, cell in enumerate(sent):\n                if isinstance(cell, tuple):","sourceCodeStart":680,"sourceCodeEnd":716,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/transform/conll_tf.py#L680-L716","documentation":"During transform.fit(), each sentence cell must follow a known arrangement (with/without POS, dependency columns). If a cell's structure matches none of the expected shapes — typically because cell has more/fewer columns than the if-branches handle — fit raises 'Unknown data arrangement'.","triggerScenarios":"Fitting a CoNLL transform on training data whose per-token tuples are neither (form,) / (form, cpos) plus (head, deprel) shapes, e.g. extra columns (lemma, feats, deps) producing cell[1:] of length > 2 or unexpected nesting.","commonSituations":"Feeding full CoNLL-U files (10 columns) into a transform written for reduced CoNLL-X subsets; changing use_pos without re-slicing the data generator; dataset format changes between versions.","solutions":["Inspect a sample sentence passed to fit and print the structure of each cell; slice your generator to keep only the columns the transform expects (form, optional cpos, head, deprel).","Ensure use_pos on the transform matches whether cells carry cpos.","If you need more columns, extend the branch logic in the transform subclass rather than feeding raw rows."],"exampleFix":"# before\ntransform.fit(conllu_rows)  # full 10-col tuples\n# after\ntrimmed = [[(tok[1], tok[4], tok[6], tok[7]) for tok in sent] for sent in conllu_rows]\ntransform.fit(trimmed)","handlingStrategy":"validation","validationCode":"for sent in data[:1]:\n    for cell in sent:\n        n = len(cell) if not isinstance(cell[0], tuple) else None\n        assert n is None or n in (2, 3), f'unexpected cell {cell}'","typeGuard":"def cell_ok(cell, use_pos):\n    form_cpos = cell[0]\n    head_dep = cell[1:]\n    ok_pair = len(form_cpos) == (2 if use_pos else 1)\n    return ok_pair and len(head_dep) == 2","tryCatchPattern":null,"preventionTips":["Slice corpora to exactly the columns the transform needs before fit().","Keep use_pos consistent with how you build token tuples; add a fit-time assert on a sample sentence."],"tags":["data-format","conll","transform","validation"],"backgroundTag":"unexpected-data-format","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}