{"record":{"id":"c98c5e18c17f0871","repo":"xai-org/x-algorithm","slug":"batch-missing-neweventmask-conversion-label-foldi","errorCode":null,"errorMessage":"batch missing newEventMask; conversion-label folding is candidate-only","messagePattern":"batch missing newEventMask; conversion-label folding is candidate-only","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"phoenix/xrex/data/conversion_labels.py","lineNumber":107,"sourceCode":"    col_idx = batch.schema.get_field_index(ACTION_MULTIHOT_COLUMN)\n    if col_idx < 0:\n        raise ValueError(f\"batch missing {ACTION_MULTIHOT_COLUMN}\")\n    mh = batch.column(col_idx)\n    seq_len = mh.type.list_size\n    vocab = mh.type.value_type.list_size\n    bits = (\n        mh.flatten()\n        .flatten()\n        .to_numpy(zero_copy_only=False)\n        .astype(np.bool_)\n        .reshape(batch.num_rows, seq_len, vocab)\n        .copy()\n    )\n    mask_name = next(\n        (n for n in (\"newEventMask\", \"newEventMaskSeq\") if n in batch.schema.names), None\n    )\n    if mask_name is None:\n        raise ValueError(\"batch missing newEventMask; conversion-label folding is candidate-only\")\n    mask_col = batch.column(mask_name)\n    is_candidate = (\n        mask_col.flatten()\n        .to_numpy(zero_copy_only=False)\n        .astype(np.bool_)\n        .reshape(batch.num_rows, mask_col.type.list_size)\n    )\n    if is_candidate.shape[1] != seq_len:\n        raise ValueError(f\"{mask_name}: seq len {is_candidate.shape[1]} != multi-hot {seq_len}\")\n    for name in action_cols:\n        idx = action_index_of(name)\n        if idx >= vocab:\n            raise ValueError(f\"{name}: bit {idx} outside vocab {vocab}\")\n        delays_col = batch.column(name)\n        delays = (\n            delays_col.flatten()\n            .to_numpy(zero_copy_only=False)\n            .astype(np.int64)","sourceCodeStart":89,"sourceCodeEnd":125,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/data/conversion_labels.py#L89-L125","documentation":"When folding conversion labels, the function needs a candidate mask to restrict folding to candidate positions. It looks for 'newEventMask' or 'newEventMaskSeq' in the batch schema; if neither exists it raises ValueError stating folding is candidate-only. Without the mask there is no safe way to decide which sequence positions may receive labels.","triggerScenarios":"Producer consuming batches from a pipeline version that predates newEventMask; upstream feature selection excluding both mask columns; delay columns attached to a hand-built RecordBatch without the mask.","commonSituations":"Schema drift after upgrading one pipeline stage but not another; datasets exported before the mask feature existed.","solutions":["Include newEventMask (or newEventMaskSeq) in the batch schema before folding.","Regenerate the dataset with a pipeline version that emits the mask.","Disable action-delay folding until the mask is available."],"exampleFix":"// before\nbatch = fold_action_delays_into_multihot(batch_no_mask, window_ms)  # ValueError\n\n// after\nbatch = fold_action_delays_into_multihot(batch_with_newEventMask, window_ms)","handlingStrategy":"validation","validationCode":"mask_name = next((n for n in ('newEventMask','newEventMaskSeq') if n in batch.schema.names), None)\nif mask_name is None:\n    disable_action_delay_folding()  # config-driven fallback","typeGuard":"def has_candidate_mask(batch) -> bool:\n    return any(n in batch.schema.names for n in ('newEventMask', 'newEventMaskSeq'))","tryCatchPattern":"try:\n    batch = fold_action_delays_into_multihot(batch, w)\nexcept ValueError as e:\n    if 'newEventMask' in str(e):\n        logger.warning('skipping fold: %s', e)\n    else:\n        raise","preventionTips":["Include the mask column in all exported training schemas.","Fail fast at startup on a sample batch schema check."],"tags":["schema","candidate-mask","conversion-labels"],"backgroundTag":"missing-required-column","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}