hankcs/HanLP · error · NotImplementedError

transformers has its own tagger, not need to convert idx for

Error message

transformers has its own tagger, not need to convert idx for y

What it means

Companion to x_to_idx: the TF transformer tagger maps labels internally, so y_to_idx is deliberately unimplemented and raises NotImplementedError when invoked. It exists only to satisfy the abstract interface of the base Tagger class.

Source

Thrown at hanlp/components/taggers/transformers/transformer_transform_tf.py:115

                                                                                         # pad on the left for xlnet
                                                                                         pad_token_id=pad_token,
                                                                                         pad_token_segment_id=4 if xlnet else 0,
                                                                                         pad_token_label_id=pad_label_idx,
                                                                                         unk_token=unk_token)

            if None in input_ids:
                print(input_ids)
            if None in input_mask:
                print(input_mask)
            if None in segment_ids:
                print(input_mask)
            yield (input_ids, input_mask, segment_ids), label_ids

    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:
        raise NotImplementedError('transformers has its own tagger, not need to convert idx for x')

    def y_to_idx(self, y) -> tf.Tensor:
        raise NotImplementedError('transformers has its own tagger, not need to convert idx for y')

    def input_is_single_sample(self, input: Union[List[str], List[List[str]]]) -> bool:
        return isinstance(input[0], str)

    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, X=None, inputs=None, batch=None,
                     **kwargs) -> Iterable:
        assert batch is not None, 'Need the batch to know actual length of Y'
        label_mask = batch[1]
        if self.tag_vocab.pad_token:
            Y[:, :, self.tag_vocab.pad_idx] = float('-inf')
        Y = tf.argmax(Y, axis=-1)
        Y = Y[label_mask > 0]
        tags = [self.tag_vocab.idx_to_token[tid] for tid in Y]
        offset = 0
        for words in inputs:
            yield tags[offset:offset + len(words)]
            offset += len(words)

View on GitHub (pinned to ddb1299bdd)

Solutions

1. Pass raw labels through the tagger's own featurization/batching path
2. In generic code, branch on the model type and skip index conversion for transformer TF taggers

Example fix

# before
y_idx = tagger.y_to_idx(labels)
# after
# labels go straight into the model's internal label mapping:
dataset = tagger.build_dataset(samples, labels)

from hanlp.components.taggers.transformers.transformer_transform_tf import TransformerTransformTagger
if isinstance(tagger, TransformerTransformTagger):
    pass  # labels handled internally; do not call y_to_idx

def needs_label_idx_conversion(tagger) -> bool:
    from hanlp.components.taggers.transformers.transformer_transform_tf import TransformerTransformTagger
    return not isinstance(tagger, TransformerTransformTagger)

When it happens

Trigger: Calling y_to_idx(y) on a TransformerTransformTagger (TF), e.g. from a generic loop that converts labels before batching, or from custom code reusing the base Tagger contract.

Common situations: Reusing generic preprocessing from the non-transformer tagger base class; building a custom training script around the label-indexing API.

Related errors

• transformers has its own tagger, not need to convert idx for
• Unsupported tagging scheme {tagging_scheme}.
• error
• output ({}) must be of type bool or str
• Call fit or load before evaluate.

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/3facdd6940f895a8. Report an issue: GitHub.