hankcs/HanLP · error · ValueError

Unknown data arrangement

Error message

Unknown data arrangement

What it means

During transform.fit(), each sentence cell must follow a known arrangement (with/without POS, dependency columns). If a cell's structure matches none of the expected shapes — typically because cell has more/fewer columns than the if-branches handle — fit raises 'Unknown data arrangement'.

Source

Thrown at hanlp/transform/conll_tf.py:698

        self.form_vocab.add(ROOT)  # make root the 2ed elements while 0th is pad, 1st is unk
        if self.use_pos:
            self.cpos_vocab = VocabTF(pad_token=None, unk_token=None)
        self.rel_vocab = VocabTF(pad_token=None, unk_token=None)
        num_samples = 0
        counter = Counter()
        for sent in self.file_to_samples(trn_path, gold=True):
            num_samples += 1
            for idx, cell in enumerate(sent):
                if len(cell) == 4:
                    form, cpos, head, deprel = cell
                elif len(cell) == 3:
                    if self.use_pos:
                        form, cpos = cell[0]
                    else:
                        form = cell[0]
                    head, deprel = cell[1:]
                else:
                    raise ValueError('Unknown data arrangement')
                if idx == 0:
                    root = form
                else:
                    counter[form] += 1
                if self.use_pos:
                    self.cpos_vocab.add(cpos)
                self.rel_vocab.update(deprel)

        for token in [token for token, freq in counter.items() if freq >= self.config.min_freq]:
            self.form_vocab.add(token)
        return num_samples

    def inputs_to_samples(self, inputs, gold=False):
        use_pos = self.use_pos
        for sent in inputs:
            sample = []
            for i, cell in enumerate(sent):
                if isinstance(cell, tuple):

View on GitHub (pinned to ddb1299bdd)

Solutions

1. Inspect a sample sentence passed to fit and print the structure of each cell; slice your generator to keep only the columns the transform expects (form, optional cpos, head, deprel).
2. Ensure use_pos on the transform matches whether cells carry cpos.
3. If you need more columns, extend the branch logic in the transform subclass rather than feeding raw rows.

Example fix

# before
transform.fit(conllu_rows)  # full 10-col tuples
# after
trimmed = [[(tok[1], tok[4], tok[6], tok[7]) for tok in sent] for sent in conllu_rows]
transform.fit(trimmed)

for sent in data[:1]:
    for cell in sent:
        n = len(cell) if not isinstance(cell[0], tuple) else None
        assert n is None or n in (2, 3), f'unexpected cell {cell}'

def cell_ok(cell, use_pos):
    form_cpos = cell[0]
    head_dep = cell[1:]
    ok_pair = len(form_cpos) == (2 if use_pos else 1)
    return ok_pair and len(head_dep) == 2

When it happens

Trigger: Fitting a CoNLL transform on training data whose per-token tuples are neither (form,) / (form, cpos) plus (head, deprel) shapes, e.g. extra columns (lemma, feats, deps) producing cell[1:] of length > 2 or unexpected nesting.

Common situations: Feeding full CoNLL-U files (10 columns) into a transform written for reduced CoNLL-X subsets; changing use_pos without re-slicing the data generator; dataset format changes between versions.

Related errors

• Failed to load {tsv_file_path}: {sent}
• the first two dimensions of emissions and tags must match, g
• Unrecognized label encoding {self.label_encoding}
• Prediction file {pred_file.name} does not end a sentence at
• Expect X to be 2 or 3 elements but got {repr(X)}

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/0e5c7f8b5e5ab2e1. Report an issue: GitHub.