deepset-ai/haystack · error · TypeError

Couldn't convert HuggingFace device map - unexpected device

Error message

Couldn't convert HuggingFace device map - unexpected device '{str(device)}' for '{key}'

What it means

DeviceMap.from_hf raises TypeError when converting a HuggingFace accelerate device_map entry whose value is neither an int, a device string, nor a torch.device. Accelerate accepts some values (like 'disk' or exotic objects) Haystack cannot map to its Device representation.

Source

Thrown at haystack/utils/device.py:241

        :param hf_device_map:
            The HuggingFace device map.
        :returns:
            The deserialized device map.
        :raises TypeError: If a device value in the map is not an int, str, or torch.device.
        """
        mapping = {}
        for key, device in hf_device_map.items():
            if isinstance(device, int):
                mapping[key] = Device(DeviceType.GPU, device)
            elif isinstance(device, str):
                device_type, device_id = _split_device_string(device)
                mapping[key] = Device(DeviceType.from_str(device_type), device_id)
            elif isinstance(device, torch.device):
                device_type = device.type
                device_id = device.index
                mapping[key] = Device(DeviceType.from_str(device_type), device_id)
            else:
                raise TypeError(
                    f"Couldn't convert HuggingFace device map - unexpected device '{str(device)}' for '{key}'"
                )
        return DeviceMap(mapping)


@dataclass(frozen=True)
class ComponentDevice:
    """
    A representation of a device for a component.

    This can be either a single device or a device map.
    """

    _single_device: Device | None = field(default=None)
    _multiple_devices: DeviceMap | None = field(default=None)

    @classmethod
    def from_str(cls, device_str: str) -> "ComponentDevice":

View on GitHub (pinned to e318778c9b)

Solutions

1. Inspect the device_map dict and normalize entries to ints or plain device strings ('cuda:0', 'cpu', 'disk') before calling from_hf
2. Recreate the device_map with explicit values instead of 'auto', e.g. {'': 'cuda:0'}
3. Upgrade haystack and/or accelerate so the device map formats match
4. Convert the map manually: build haystack DeviceMap from parsed Device entries

Example fix

// before
comp = ComponentDevice.from_hf(device_map=auto_map)  # entries like {'model.layers': 'meta'}
// after
device_map = {'': 'cuda:0'}
comp = ComponentDevice.from_hf(device_map=device_map)

ALLOWED = (int, str, torch.device) if torch else (int, str)
def is_hf_convertible(device_map: dict) -> bool:
    return all(isinstance(v, ALLOWED) for v in device_map.values())

def is_convertible_device(v: object) -> bool:
    return isinstance(v, (int, str)) or (torch and isinstance(v, torch.device))

try:
    comp = ComponentDevice.from_hf(device_map)
except TypeError as e:
    print(f"Device map not convertible: {e}")
    comp = ComponentDevice.from_str("cpu")

When it happens

Trigger: Calling ComponentDevice.from_hf(device_map=...) with an accelerate device_map containing unexpected per-layer values, e.g. values of an unsupported type, empty/None entries, or device strings Haystack cannot parse.

Common situations: Loading a large model with device_map='auto' from accelerate and passing the resulting mapping into Haystack; accelerate produced an entry referencing storage Haystack does not model; version mismatch between accelerate and haystack's device handling.

Related errors

• Hook registered for hook point '{hook_point}' is callable bu
• {type(self.chat_generator).__name__} does not accept tools p
• The {self.__class__.__name__} expects a list containing only
• Unsupported source type {type(source)}
• meta must be either None, a dictionary or a list of dictiona

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/c74890d6f6a47df4. Report an issue: GitHub.