Note: the "to_float" is True, you need to ensure that the…

Error message

Note: the "to_float" is True, you need to ensure that the behavior is reasonable.

What it means

Inside all_reduce_dict, when to_float=True (the default) the code warns that every tensor in the dict will be cast to float32 before the all-reduce and concatenated. This matters for integer-valued metrics: casting to float can lose precision for very large counts (beyond 2^24) and changes dtypes of the returned values, so the author asks you to confirm that is reasonable for your data.

Solutions

If your values are counts or integers, call all_reduce_dict(metrics, to_float=False) to keep original dtypes.
If your values are losses/accuracies (small floats), the default is fine — filter the warning: warnings.filterwarnings('ignore', message='.*to_float.*').
For large counts, reduce in chunks or scale down (e.g. count / world_size) if you must keep to_float=True.
Verify returned dtypes after reduction before doing integer-sensitive arithmetic.

Example fix

# before
reduced = all_reduce_dict(metrics)  # to_float=True default, warns
# after
reduced = all_reduce_dict(metrics, to_float=False)  # preserve original dtypes

Defensive patterns

Strategy: validation

Validate before calling

import torch

def metrics_are_float_safe(py_dict) -> bool:
    return all(
        torch.is_tensor(v) and (not v.dtype.is_floating_point or v.abs().max() < 2**24)
        for v in py_dict.values()
    )

Prevention

Pass to_float=False when reducing integer counts or large-magnitude tensors.
Keep to_float=True only for losses/rates (small floats).
Check dtypes of the returned dict before integer-sensitive arithmetic.

When it happens

Trigger: Calling all_reduce_dict(py_dict) without to_float=False — i.e. using the default — during distributed validation metric aggregation. The warning fires whenever world_size > 1 path is taken (before the flatten/cast) or even earlier in single-process after the group notice.

Common situations: Aggregating validation losses/mAP values (floats — fine, warning ignorable); aggregating raw sample counts or pixel counts across many ranks where values exceed float32 integer precision; code that later does exact integer comparisons on the reduced values.

Related errors

AI-assisted analysis of open-mmlab/mmdetection@cfd5d3a985 (2026-08-27). Data as JSON: /api/errors/3585cd6fb86fd442. Report an issue: GitHub.

Appendix: source

Thrown at mmdet/utils/dist_utils.py:127

    """
    warnings.warn(
        'group` is deprecated. Currently only supports NCCL backend.')
    _, world_size = get_dist_info()
    if world_size == 1:
        return py_dict

    # all reduce logic across different devices.
    py_key = list(py_dict.keys())
    if not isinstance(py_dict, OrderedDict):
        py_key_tensor = obj2tensor(py_key)
        dist.broadcast(py_key_tensor, src=0)
        py_key = tensor2obj(py_key_tensor)

    tensor_shapes = [py_dict[k].shape for k in py_key]
    tensor_numels = [py_dict[k].numel() for k in py_key]

    if to_float:
        warnings.warn('Note: the "to_float" is True, you need to '
                      'ensure that the behavior is reasonable.')
        flatten_tensor = torch.cat(
            [py_dict[k].flatten().float() for k in py_key])
    else:
        flatten_tensor = torch.cat([py_dict[k].flatten() for k in py_key])

    dist.all_reduce(flatten_tensor, op=dist.ReduceOp.SUM)
    if op == 'mean':
        flatten_tensor /= world_size

    split_tensors = [
        x.reshape(shape) for x, shape in zip(
            torch.split(flatten_tensor, tensor_numels), tensor_shapes)
    ]
    out_dict = {k: v for k, v in zip(py_key, split_tensors)}
    if isinstance(py_dict, OrderedDict):
        out_dict = OrderedDict(out_dict)
    return out_dict

View on GitHub (pinned to cfd5d3a985)