hankcs/HanLP · error · NotImplementedError

transformers has its own tagger, not need to convert idx for

Error message

transformers has its own tagger, not need to convert idx for x

What it means

Transformer taggers in the TF backend do their own tokenization/indexing inside the model, so the generic x_to_idx conversion hook is intentionally not implemented and always raises NotImplementedError. It is a guard against calling the base-class preprocessing API on a model that bypasses it.

Source

Thrown at hanlp/components/taggers/transformers/transformer_transform_tf.py:112

                                                                                         sep_token_extra=roberta,
                                                                                         # roberta uses an extra separator b/w pairs of sentences, cf. github.com/pytorch/fairseq/commit/1684e166e3da03f5b600dbb7855cb98ddfcd0805
                                                                                         pad_on_left=xlnet,
                                                                                         # pad on the left for xlnet
                                                                                         pad_token_id=pad_token,
                                                                                         pad_token_segment_id=4 if xlnet else 0,
                                                                                         pad_token_label_id=pad_label_idx,
                                                                                         unk_token=unk_token)

            if None in input_ids:
                print(input_ids)
            if None in input_mask:
                print(input_mask)
            if None in segment_ids:
                print(input_mask)
            yield (input_ids, input_mask, segment_ids), label_ids

    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:
        raise NotImplementedError('transformers has its own tagger, not need to convert idx for x')

    def y_to_idx(self, y) -> tf.Tensor:
        raise NotImplementedError('transformers has its own tagger, not need to convert idx for y')

    def input_is_single_sample(self, input: Union[List[str], List[List[str]]]) -> bool:
        return isinstance(input[0], str)

    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, X=None, inputs=None, batch=None,
                     **kwargs) -> Iterable:
        assert batch is not None, 'Need the batch to know actual length of Y'
        label_mask = batch[1]
        if self.tag_vocab.pad_token:
            Y[:, :, self.tag_vocab.pad_idx] = float('-inf')
        Y = tf.argmax(Y, axis=-1)
        Y = Y[label_mask > 0]
        tags = [self.tag_vocab.idx_to_token[tid] for tid in Y]
        offset = 0
        for words in inputs:

View on GitHub (pinned to ddb1299bdd)

Solutions

1. Do not call x_to_idx for transformer TF taggers; the model tokenizes raw text internally
2. Build training examples via the class's generator/featurization methods (e.g. its input_fn / generate_batches path) instead
3. If writing generic code, skip these hooks for transformers subclasses (check isinstance)

Example fix

# before
idx = tagger.x_to_idx(samples)
# after
# transformer tagger consumes raw samples directly:
examples = tagger.generate_instances(samples)

assert not hasattr(tagger, 'x_to_idx') and callable(getattr(type(tagger), 'x_to_idx', None)) and not getattr(type(tagger).x_to_idx, '__isabstractmethod__', False)
# simpler: skip manual idx conversion for transformer TF taggers
from hanlp.components.taggers.transformers.transformer_transform_tf import TransformerTransformTagger
manual_convert = not isinstance(tagger, TransformerTransformTagger)

def needs_idx_conversion(tagger) -> bool:
    from hanlp.components.taggers.transformers.transformer_transform_tf import TransformerTransformTagger
    return not isinstance(tagger, TransformerTransformTagger)

When it happens

Trigger: Calling x_to_idx(x) on a TransformerTransformTagger (TF) instance, typically via generic training/prediction code inherited from Tagger that assumes the hook exists, or manual use of the preprocessing API.

Common situations: Writing generic preprocessing pipelines that call x_to_idx/y_to_idx on any tagger; subclassing the TF transformer tagger and accidentally invoking the parent's index-building path.

Related errors

• transformers has its own tagger, not need to convert idx for
• Unsupported tagging scheme {tagging_scheme}.
• error
• output ({}) must be of type bool or str
• Call fit or load before evaluate.

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/00fc77cff6c497d0. Report an issue: GitHub.