{"record":{"id":"a92b955ea54957dc","repo":"xai-org/x-algorithm","slug":"fused-lazy-decay-needs-timestamped-state-step-las","errorCode":null,"errorMessage":"fused lazy decay needs timestamped state (step/last_step)","messagePattern":"fused lazy decay needs timestamped state \\(step/last_step\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"phoenix/xrex/optimizers/recsys/rowwise_adagrad.py","lineNumber":103,"sourceCode":"        token_ids: jax.Array,\n        state: Any,\n    ) -> tuple[Any, Any]:\n        return embeddings, None\n\n    @property\n    def decay_factor(self) -> float:\n        return math.exp(-self._decay_rate) if self._decay_rate is not None else 1.0\n\n    def gradient_update_start(\n        self,\n        context: async_emb.AsyncEmbContextHandle,\n        update: AsyncEmbGradientUpdate,\n        table: jax.Array,\n        state: RecsysRowwiseAdagradState,\n        gate: jax.Array,\n    ) -> tuple[tuple[jax.Array, ...], jax.Array, RecsysRowwiseAdagradState]:\n        if self._lazy_decay and (state.step is None or state.last_step is None):\n            raise ValueError(\"fused lazy decay needs timestamped state (step/last_step)\")\n\n        if 32 % context.shard_width != 0:\n            raise ValueError(\n                f\"the fused rowwise Adagrad update needs a row shard that divides a warp, \"\n                f\"got shard_width={context.shard_width} (emb_width={context.emb_width})\"\n            )\n\n        from xrex.cuda.async_emb import async_emb\n\n        metrics: dict[str, jax.Array] = {}\n\n        if self._lazy_decay:\n\n            @shard_map(\n                mesh=context.mesh,\n                in_specs=(\n                    P(context.data_axis, None),\n                    P(context.data_axis),","sourceCodeStart":85,"sourceCodeEnd":121,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/optimizers/recsys/rowwise_adagrad.py#L85-L121","documentation":"Thrown by the fused rowwise Adagrad embedding optimizer when lazy decay (decaying stale embedding rows only when they are touched) is enabled but the optimizer state lacks step counters. The fused CUDA kernel applies per-row decay based on the difference between the global step and each row's last-updated step, so both state.step and state.last_step must be present.","triggerScenarios":"Calling gradient_update_start with _lazy_decay=True while the RecsysRowwiseAdagradState was constructed without step/last_step fields (e.g. a freshly initialized or deserialized state that omits timestamps).","commonSituations":"Enabling lazy decay in a recsys training config after previously running without it, reusing old checkpoints whose optimizer state predates timestamping, or building the state manually in tests.","solutions":["Initialize the optimizer state with step counters (e.g. zeros for last_step and a step counter array) before enabling _lazy_decay","If timestamped state is unavailable, disable _lazy_decay and use dense decay instead","Migrate/regenerate the checkpoint so it includes step and last_step fields"],"exampleFix":"# before\nopt = RecsysRowwiseAdagrad(lazy_decay=True)\n# state = opt.init_state(...)  # no step/last_step\n\n# after\nopt = RecsysRowwiseAdagrad(lazy_decay=True)\nstate = opt.init_state(..., with_step_counters=True)  # state.step / state.last_step populated","handlingStrategy":"validation","validationCode":"if opt._lazy_decay and (state.step is None or state.last_step is None):\n    raise ValueError(\"initialize step/last_step before enabling lazy decay\")","typeGuard":"def has_timestamps(s: RecsysRowwiseAdagradState) -> bool:\n    return s.step is not None and s.last_step is not None","tryCatchPattern":null,"preventionTips":["Always create optimizer state via opt.init_state rather than hand-constructing it","Add a startup assert that lazy_decay implies timestamped state"],"tags":["recsys","adagrad","optimizer","embedding","lazy-decay"],"backgroundTag":"optimizer-state-mismatch","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}