Lightning-AI/pytorch-lightning · error · NotImplementedError

The Kubeflow environment can't be detected automatically.

Error message

The Kubeflow environment can't be detected automatically.

What it means

`KubeflowEnvironment.detect()` deliberately raises NotImplementedError because the Kubeflow MPI environment cannot be auto-detected from ambient signals — Kubeflow sets the same env vars (RANK, WORLD_SIZE) as other launchers, making detection ambiguous. You must explicitly instantiate/choose this ClusterEnvironment.

Source

Thrown at src/lightning/fabric/plugins/environments/kubeflow.py:54

    @property
    @override
    def creates_processes_externally(self) -> bool:
        return True

    @property
    @override
    def main_address(self) -> str:
        return os.environ["MASTER_ADDR"]

    @property
    @override
    def main_port(self) -> int:
        return int(os.environ["MASTER_PORT"])

    @staticmethod
    @override
    def detect() -> bool:
        raise NotImplementedError("The Kubeflow environment can't be detected automatically.")

    @override
    def world_size(self) -> int:
        return int(os.environ["WORLD_SIZE"])

    @override
    def set_world_size(self, size: int) -> None:
        log.debug("KubeflowEnvironment.set_world_size was called, but setting world size is not allowed. Ignored.")

    @override
    def global_rank(self) -> int:
        return int(os.environ["RANK"])

    @override
    def set_global_rank(self, rank: int) -> None:
        log.debug("KubeflowEnvironment.set_global_rank was called, but setting global rank is not allowed. Ignored.")

    @override

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Pass the environment explicitly: `KubeflowEnvironment()` to your strategy/accelerator (e.g. `DDPStrategy(cluster_environment=KubeflowEnvironment())` or the Fabric equivalents)
2. Never rely on auto-detection inside Kubeflow; select the environment by config
3. If writing generic detection loops, skip environments whose detect raises NotImplementedError

Example fix

# before
env = next(e for e in envs if e.detect())  # hits KubeflowEnvironment.detect -> NotImplementedError

# after
from lightning.fabric.plugins.environments import KubeflowEnvironment
env = KubeflowEnvironment()  # explicit selection

from lightning.fabric.plugins.environments import KubeflowEnvironment
env = KubeflowEnvironment()  # never call .detect() on it

try:
    ok = env_cls.detect()
except NotImplementedError:
    ok = False  # requires explicit selection

When it happens

Trigger: Running launcher auto-detection that calls `detect()` on `KubeflowEnvironment`, e.g. letting Lightning infer the cluster environment in a Kubeflow MPIJob pod, or calling `KubeflowEnvironment.detect()` manually.

Common situations: Training inside Kubeflow/MPIOperator pods where users expect automatic environment detection; custom launcher logic that iterates over all environment classes calling detect().

Related errors

• `{type(self).__name__}` does not own a group. HINT: try `col
• `{type(self).__name__}` already owns a group.
• `{type(self).__name__}` does not own a group to destroy.
• Torch distributed is not available.
• Unsupported op {op!r} of type {type(op).__name__}

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/eee9c1572e5275a2. Report an issue: GitHub.