{"record":{"id":"3facdd6940f895a8","repo":"hankcs/HanLP","slug":"transformers-has-its-own-tagger-not-need-to-conve-3facdd","errorCode":null,"errorMessage":"transformers has its own tagger, not need to convert idx for y","messagePattern":"transformers has its own tagger, not need to convert idx for y","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"hanlp/components/taggers/transformers/transformer_transform_tf.py","lineNumber":115,"sourceCode":"                                                                                         # pad on the left for xlnet\n                                                                                         pad_token_id=pad_token,\n                                                                                         pad_token_segment_id=4 if xlnet else 0,\n                                                                                         pad_token_label_id=pad_label_idx,\n                                                                                         unk_token=unk_token)\n\n            if None in input_ids:\n                print(input_ids)\n            if None in input_mask:\n                print(input_mask)\n            if None in segment_ids:\n                print(input_mask)\n            yield (input_ids, input_mask, segment_ids), label_ids\n\n    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:\n        raise NotImplementedError('transformers has its own tagger, not need to convert idx for x')\n\n    def y_to_idx(self, y) -> tf.Tensor:\n        raise NotImplementedError('transformers has its own tagger, not need to convert idx for y')\n\n    def input_is_single_sample(self, input: Union[List[str], List[List[str]]]) -> bool:\n        return isinstance(input[0], str)\n\n    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, X=None, inputs=None, batch=None,\n                     **kwargs) -> Iterable:\n        assert batch is not None, 'Need the batch to know actual length of Y'\n        label_mask = batch[1]\n        if self.tag_vocab.pad_token:\n            Y[:, :, self.tag_vocab.pad_idx] = float('-inf')\n        Y = tf.argmax(Y, axis=-1)\n        Y = Y[label_mask > 0]\n        tags = [self.tag_vocab.idx_to_token[tid] for tid in Y]\n        offset = 0\n        for words in inputs:\n            yield tags[offset:offset + len(words)]\n            offset += len(words)\n","sourceCodeStart":97,"sourceCodeEnd":133,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/components/taggers/transformers/transformer_transform_tf.py#L97-L133","documentation":"Companion to x_to_idx: the TF transformer tagger maps labels internally, so y_to_idx is deliberately unimplemented and raises NotImplementedError when invoked. It exists only to satisfy the abstract interface of the base Tagger class.","triggerScenarios":"Calling y_to_idx(y) on a TransformerTransformTagger (TF), e.g. from a generic loop that converts labels before batching, or from custom code reusing the base Tagger contract.","commonSituations":"Reusing generic preprocessing from the non-transformer tagger base class; building a custom training script around the label-indexing API.","solutions":["Pass raw labels through the tagger's own featurization/batching path","In generic code, branch on the model type and skip index conversion for transformer TF taggers"],"exampleFix":"# before\ny_idx = tagger.y_to_idx(labels)\n# after\n# labels go straight into the model's internal label mapping:\ndataset = tagger.build_dataset(samples, labels)","handlingStrategy":"validation","validationCode":"from hanlp.components.taggers.transformers.transformer_transform_tf import TransformerTransformTagger\nif isinstance(tagger, TransformerTransformTagger):\n    pass  # labels handled internally; do not call y_to_idx","typeGuard":"def needs_label_idx_conversion(tagger) -> bool:\n    from hanlp.components.taggers.transformers.transformer_transform_tf import TransformerTransformTagger\n    return not isinstance(tagger, TransformerTransformTagger)","tryCatchPattern":null,"preventionTips":["Prefer the class-provided batch-generation APIs over manual label indexing"],"tags":["python","tensorflow","transformers","not-implemented"],"backgroundTag":"unsupported-api-call","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}