Unity-Technologies/ml-agents · critical · UnityTrainerException

Inf found

Error message

Inf found

What it means

UnityTrainerException raised in sac_value_loss when the aggregated critic value loss contains Inf or NaN. After computing MSE losses between value estimates and backup targets for each reward signal, the optimizer checks torch.isinf/torch.isnan on the summed value loss and aborts the update to prevent corrupting network weights.

Source

Thrown at ml-agents/mlagents/trainers/sac/optimizer_torch.py:366

                with torch.no_grad():
                    v_backup = min_policy_qs[name] - torch.mean(
                        branched_ent_bonus, axis=0
                    )
                    # Add continuous entropy bonus to minimum Q
                    if self._action_spec.continuous_size > 0:
                        v_backup += torch.sum(
                            _cont_ent_coef * log_probs.continuous_tensor,
                            dim=1,
                            keepdim=True,
                        )
                value_loss = 0.5 * ModelUtils.masked_mean(
                    torch.nn.functional.mse_loss(values[name], v_backup.squeeze()),
                    loss_masks,
                )
                value_losses.append(value_loss)
        value_loss = torch.mean(torch.stack(value_losses))
        if torch.isinf(value_loss).any() or torch.isnan(value_loss).any():
            raise UnityTrainerException("Inf found")
        return value_loss

    def sac_policy_loss(
        self,
        log_probs: ActionLogProbs,
        q1p_outs: Dict[str, torch.Tensor],
        loss_masks: torch.Tensor,
    ) -> torch.Tensor:
        _cont_ent_coef, _disc_ent_coef = (
            self._log_ent_coef.continuous,
            self._log_ent_coef.discrete,
        )
        _cont_ent_coef = _cont_ent_coef.exp()
        _disc_ent_coef = _disc_ent_coef.exp()

        mean_q1 = torch.mean(torch.stack(list(q1p_outs.values())), axis=0)
        batch_policy_loss = 0
        if self._action_spec.discrete_size > 0:

View on GitHub (pinned to 3ecb446f75)

Solutions

  1. Lower the learning rate and/or add/keep gradient clipping in SAC hyperparameters to stop divergence.
  2. Scale down reward_signal strengths so value targets stay in a reasonable range.
  3. Check reward function output for Inf/NaN before training; sanitize the environment's rewards.
  4. Restart from an earlier checkpoint (via --init-file) taken before the value loss blew up.

Example fix

# before
rewards:
  extrinsic:
    strength: 100.0
# after
rewards:
  extrinsic:
    strength: 1.0
Defensive patterns

Strategy: validation

Validate before calling

import numpy as np
assert np.isfinite(env_reward).all(), "Environment emits non-finite rewards"
assert all(0 < s <= 10 for s in reward_signal_strengths), "Reward strengths too large"

Type guard

import torch
def value_loss_is_finite(loss: torch.Tensor) -> bool:
    return bool(torch.isfinite(loss).all())

Try / catch

from mlagents.trainers.exception import UnityTrainerException
try:
    optimizer.update(batch, num_sequences)
except UnityTrainerException as e:
    if "Inf found" in str(e):
        logger.error("SAC value loss diverged; loading last stable checkpoint")
        policy.load(last_good_checkpoint)

Prevention

When it happens

Trigger: During SAC update() when value targets explode — e.g. extremely large rewards/reward signal strengths, discount factor near 1 with non-terminal bootstrap, buffer of bad data, or the Q/value networks already containing NaN from a previous diverged update.

Common situations: Reward signals with huge magnitudes (e.g. unscaled extrinsic + curiosity); learning rate too high causing divergence mid-training; reward signal strength settings producing backup values in the thousands+.

Related errors

AI-assisted analysis of Unity-Technologies/ml-agents@3ecb446f75 (2026-09-02). Data as JSON: /api/errors/41d75adfc3d48fe3. Report an issue: GitHub.