{"record":{"id":"00fc77cff6c497d0","repo":"hankcs/HanLP","slug":"transformers-has-its-own-tagger-not-need-to-conve","errorCode":null,"errorMessage":"transformers has its own tagger, not need to convert idx for x","messagePattern":"transformers has its own tagger, not need to convert idx for x","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"hanlp/components/taggers/transformers/transformer_transform_tf.py","lineNumber":112,"sourceCode":"                                                                                         sep_token_extra=roberta,\n                                                                                         # roberta uses an extra separator b/w pairs of sentences, cf. github.com/pytorch/fairseq/commit/1684e166e3da03f5b600dbb7855cb98ddfcd0805\n                                                                                         pad_on_left=xlnet,\n                                                                                         # pad on the left for xlnet\n                                                                                         pad_token_id=pad_token,\n                                                                                         pad_token_segment_id=4 if xlnet else 0,\n                                                                                         pad_token_label_id=pad_label_idx,\n                                                                                         unk_token=unk_token)\n\n            if None in input_ids:\n                print(input_ids)\n            if None in input_mask:\n                print(input_mask)\n            if None in segment_ids:\n                print(input_mask)\n            yield (input_ids, input_mask, segment_ids), label_ids\n\n    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:\n        raise NotImplementedError('transformers has its own tagger, not need to convert idx for x')\n\n    def y_to_idx(self, y) -> tf.Tensor:\n        raise NotImplementedError('transformers has its own tagger, not need to convert idx for y')\n\n    def input_is_single_sample(self, input: Union[List[str], List[List[str]]]) -> bool:\n        return isinstance(input[0], str)\n\n    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, X=None, inputs=None, batch=None,\n                     **kwargs) -> Iterable:\n        assert batch is not None, 'Need the batch to know actual length of Y'\n        label_mask = batch[1]\n        if self.tag_vocab.pad_token:\n            Y[:, :, self.tag_vocab.pad_idx] = float('-inf')\n        Y = tf.argmax(Y, axis=-1)\n        Y = Y[label_mask > 0]\n        tags = [self.tag_vocab.idx_to_token[tid] for tid in Y]\n        offset = 0\n        for words in inputs:","sourceCodeStart":94,"sourceCodeEnd":130,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/components/taggers/transformers/transformer_transform_tf.py#L94-L130","documentation":"Transformer taggers in the TF backend do their own tokenization/indexing inside the model, so the generic x_to_idx conversion hook is intentionally not implemented and always raises NotImplementedError. It is a guard against calling the base-class preprocessing API on a model that bypasses it.","triggerScenarios":"Calling x_to_idx(x) on a TransformerTransformTagger (TF) instance, typically via generic training/prediction code inherited from Tagger that assumes the hook exists, or manual use of the preprocessing API.","commonSituations":"Writing generic preprocessing pipelines that call x_to_idx/y_to_idx on any tagger; subclassing the TF transformer tagger and accidentally invoking the parent's index-building path.","solutions":["Do not call x_to_idx for transformer TF taggers; the model tokenizes raw text internally","Build training examples via the class's generator/featurization methods (e.g. its input_fn / generate_batches path) instead","If writing generic code, skip these hooks for transformers subclasses (check isinstance)"],"exampleFix":"# before\nidx = tagger.x_to_idx(samples)\n# after\n# transformer tagger consumes raw samples directly:\nexamples = tagger.generate_instances(samples)","handlingStrategy":"validation","validationCode":"assert not hasattr(tagger, 'x_to_idx') and callable(getattr(type(tagger), 'x_to_idx', None)) and not getattr(type(tagger).x_to_idx, '__isabstractmethod__', False)\n# simpler: skip manual idx conversion for transformer TF taggers\nfrom hanlp.components.taggers.transformers.transformer_transform_tf import TransformerTransformTagger\nmanual_convert = not isinstance(tagger, TransformerTransformTagger)","typeGuard":"def needs_idx_conversion(tagger) -> bool:\n    from hanlp.components.taggers.transformers.transformer_transform_tf import TransformerTransformTagger\n    return not isinstance(tagger, TransformerTransformTagger)","tryCatchPattern":null,"preventionTips":["Use each tagger class's own featurization entry points rather than generic preprocessing hooks"],"tags":["python","tensorflow","transformers","not-implemented"],"backgroundTag":"unsupported-api-call","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}