{"record":{"id":"2c1cf47d1658bca6","repo":"hankcs/HanLP","slug":"got-average-f-average-expected-one-of-none-tok","errorCode":null,"errorMessage":"Got average f{average}, expected one of None, 'token', or 'batch'","messagePattern":"Got average f(.+?), expected one of None, 'token', or 'batch'","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"hanlp/components/parsers/ud/udify_util.py","lineNumber":70,"sourceCode":"        train_file = [file for file in conllu_files if file.endswith(\"train.conllu\")]\n        dev_file = [file for file in conllu_files if file.endswith(\"dev.conllu\")]\n        test_file = [file for file in conllu_files if file.endswith(\"test.conllu\")]\n\n        train_file = os.path.join(treebank_path, train_file[0]) if train_file else None\n        dev_file = os.path.join(treebank_path, dev_file[0]) if dev_file else None\n        test_file = os.path.join(treebank_path, test_file[0]) if test_file else None\n\n        datasets[treebank] = (train_file, dev_file, test_file)\n    return datasets\n\n\ndef sequence_cross_entropy(log_probs: torch.FloatTensor,\n                           targets: torch.LongTensor,\n                           weights: torch.FloatTensor,\n                           average: str = \"batch\",\n                           label_smoothing: float = None) -> torch.FloatTensor:\n    if average not in {None, \"token\", \"batch\"}:\n        raise ValueError(\"Got average f{average}, expected one of \"\n                         \"None, 'token', or 'batch'\")\n    # shape : (batch * sequence_length, num_classes)\n    log_probs_flat = log_probs.view(-1, log_probs.size(2))\n    # shape : (batch * max_len, 1)\n    targets_flat = targets.view(-1, 1).long()\n\n    if label_smoothing is not None and label_smoothing > 0.0:\n        num_classes = log_probs.size(-1)\n        smoothing_value = label_smoothing / num_classes\n        # Fill all the correct indices with 1 - smoothing value.\n        one_hot_targets = torch.zeros_like(log_probs_flat).scatter_(-1, targets_flat, 1.0 - label_smoothing)\n        smoothed_targets = one_hot_targets + smoothing_value\n        negative_log_likelihood_flat = - log_probs_flat * smoothed_targets\n        negative_log_likelihood_flat = negative_log_likelihood_flat.sum(-1, keepdim=True)\n    else:\n        # Contribution to the negative log likelihood only comes from the exact indices\n        # of the targets, as the target distributions are one-hot. Here we use torch.gather\n        # to extract the indices of the num_classes dimension which contribute to the loss.","sourceCodeStart":52,"sourceCodeEnd":88,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/components/parsers/ud/udify_util.py#L52-L88","documentation":"sequence_cross_entropy validates its `average` argument, which controls how the per-token loss is aggregated (None = sum, 'token' = per-token mean, 'batch' = per-sequence mean). Any other string (e.g. 'sentence', 'sample', or a typo like 'Batch') is rejected. This mirrors AllenNLP's sequence_cross_entropy_with_logits helper that HanLP vendors for the UD parser (udify).","triggerScenarios":"Calling sequence_cross_entropy (directly or via udify's _adaptive_loss) with average set to anything outside {None, 'token', 'batch'}, e.g. passing a config value like 'epoch' or a misspelled 'tokem'.","commonSituations":"Customizing the loss aggregation in a copied udify training script; passing an aggregation string from another library (e.g. Keras 'auto' or sklearn conventions) into this function.","solutions":["Set average to one of None, 'token', or 'batch' (default is 'batch')","If you need per-sentence normalization, use 'batch'; for per-token use 'token'; for unnormalized sum use None","Check for typos in the value passed by your config/CLI"],"exampleFix":"// before\nloss = sequence_cross_entropy(log_probs, targets, weights, average='samples')\n// after\nloss = sequence_cross_entropy(log_probs, targets, weights, average='batch')","handlingStrategy":"validation","validationCode":"assert average in (None, 'token', 'batch'), f\"bad average: {average!r}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate config strings once at startup rather than deep in the training loop"],"tags":["python","pytorch","loss-function","argument-validation"],"backgroundTag":"invalid-argument-value","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}