hankcs/HanLP · error · NotImplementedError

transformers has its own tagger, not need to convert idx…

Error message

transformers has its own tagger, not need to convert idx for y

What it means

Companion to x_to_idx: the TF transformer tagger maps labels internally, so y_to_idx is deliberately unimplemented and raises NotImplementedError when invoked. It exists only to satisfy the abstract interface of the base Tagger class.

Solutions

1. Pass raw labels through the tagger's own featurization/batching path
2. In generic code, branch on the model type and skip index conversion for transformer TF taggers

Example fix

# before
y_idx = tagger.y_to_idx(labels)
# after
# labels go straight into the model's internal label mapping:
dataset = tagger.build_dataset(samples, labels)

from hanlp.components.taggers.transformers.transformer_transform_tf import TransformerTransformTagger
if isinstance(tagger, TransformerTransformTagger):
    pass  # labels handled internally; do not call y_to_idx

def needs_label_idx_conversion(tagger) -> bool:
    from hanlp.components.taggers.transformers.transformer_transform_tf import TransformerTransformTagger
    return not isinstance(tagger, TransformerTransformTagger)

When it happens

Trigger: Calling y_to_idx(y) on a TransformerTransformTagger (TF), e.g. from a generic loop that converts labels before batching, or from custom code reusing the base Tagger contract.

Common situations: Reusing generic preprocessing from the non-transformer tagger base class; building a custom training script around the label-indexing API.

Related errors

• transformers has its own tagger, not need to convert idx…
• Unsupported tagging scheme
• activation must be callable: type=
• alpha must be float, list of float, or torch.FloatTensor
• Call fit or load before evaluate.

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/3facdd6940f895a8. Report an issue: GitHub.

Appendix: source

Thrown at hanlp/components/taggers/transformers/transformer_transform_tf.py:115

                                                                                         # pad on the left for xlnet
                                                                                         pad_token_id=pad_token,
                                                                                         pad_token_segment_id=4 if xlnet else 0,
                                                                                         pad_token_label_id=pad_label_idx,
                                                                                         unk_token=unk_token)

            if None in input_ids:
                print(input_ids)
            if None in input_mask:
                print(input_mask)
            if None in segment_ids:
                print(input_mask)
            yield (input_ids, input_mask, segment_ids), label_ids

    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:
        raise NotImplementedError('transformers has its own tagger, not need to convert idx for x')

    def y_to_idx(self, y) -> tf.Tensor:
        raise NotImplementedError('transformers has its own tagger, not need to convert idx for y')

    def input_is_single_sample(self, input: Union[List[str], List[List[str]]]) -> bool:
        return isinstance(input[0], str)

    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, X=None, inputs=None, batch=None,
                     **kwargs) -> Iterable:
        assert batch is not None, 'Need the batch to know actual length of Y'
        label_mask = batch[1]
        if self.tag_vocab.pad_token:
            Y[:, :, self.tag_vocab.pad_idx] = float('-inf')
        Y = tf.argmax(Y, axis=-1)
        Y = Y[label_mask > 0]
        tags = [self.tag_vocab.idx_to_token[tid] for tid in Y]
        offset = 0
        for words in inputs:
            yield tags[offset:offset + len(words)]
            offset += len(words)

View on GitHub (pinned to ddb1299bdd)