{"record":{"id":"1f8bd3ed38c49c51","repo":"commaai/openpilot","slug":"failed-to-open-isd-r-after-retries","errorCode":null,"errorMessage":"Failed to open ISD-R after retries","messagePattern":"Failed to open ISD-R after retries","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"openpilot/common/esim/lpa.py","lineNumber":237,"sourceCode":"  def _reset_modem(self) -> None:\n    if self._serial:\n      try:\n        self._serial.close()\n      except Exception:\n        pass\n      self._serial = None\n    subprocess.run(['/usr/comma/lte/lte.sh', 'start'], capture_output=True)\n\n  def open_isdr(self) -> None:\n    for attempt in range(OPEN_ISDR_RETRIES):\n      try:\n        self._open_isdr_once()\n        return\n      except (RuntimeError, TimeoutError, termios.error, SerialException):\n        time.sleep(OPEN_ISDR_RETRY_DELAY_S)\n        if attempt == OPEN_ISDR_RESET_ATTEMPT:\n          self._reset_modem()\n    raise RuntimeError(\"Failed to open ISD-R after retries\")\n\n  def send_apdu(self, apdu: bytes) -> tuple[bytes, int, int]:\n    for attempt in range(SEND_APDU_RETRIES):\n      try:\n        if not self.channel:\n          self.open_isdr()\n        hex_payload = apdu.hex().upper()\n        for line in self.query(f'AT+CGLA={self.channel},{len(hex_payload)},\"{hex_payload}\"'):\n          if line.startswith(\"+CGLA:\"):\n            parts = line.split(\":\", 1)[1].split(\",\", 1)\n            if len(parts) == 2:\n              data = bytes.fromhex(parts[1].strip().strip('\"'))\n              if len(data) >= 2:\n                return data[:-2], data[-2], data[-1]\n        raise RuntimeError(\"Missing +CGLA response\")\n      except (RuntimeError, ValueError):\n        self.channel = None\n        if attempt == SEND_APDU_RETRIES - 1:","sourceCodeStart":219,"sourceCodeEnd":255,"githubUrl":"https://github.com/commaai/openpilot/blob/516ec1e68203439a73f340f1d0b3b91eabc626ee/openpilot/common/esim/lpa.py#L219-L255","documentation":"open_isdr() tries to open a logical channel to the eUICC's ISD-R (ISDR_AID) up to OPEN_ISDR_RETRIES=10 times, sleeping 0.25s between attempts and resetting the modem at attempt 5. If every attempt fails with RuntimeError/TimeoutError/termios.error/SerialException, it gives up with this RuntimeError.","triggerScenarios":"Sustained modem failure: serial port unreadable (termios/SerialException), modem repeatedly answering ERROR/timeout to the channel-open AT sequence, or modem absent. Happens during heavy eSIM use or when the modem crashed earlier and lte.sh restart doesn't recover it.","commonSituations":"Modem firmware crash requiring full power cycle, not just lte.sh restart; /dev/modem_at0 permissions or udev breakage after update; no eUICC so the SELECT of ISDR AID always fails; hardware fault in the cellular module.","solutions":["Full power-cycle the device/modem (not just lte.sh start) — many ISD-R open failures need a cold reset","Verify the serial device exists and is readable/writable: ls -l /dev/modem_at0; run with DEBUG=1 to see the raw AT exchange","Confirm an eUICC is actually present (is_euicc()) before attempting ISD-R operations","Check modem firmware/health logs (dmesg, modem manager logs) for a crashed or rebooting modem; reflash/update firmware if it keeps crashing"],"exampleFix":"# before\nclient.open_isdr()  # RuntimeError: Failed to open ISD-R after retries\n\n# after\nimport subprocess\nclient._reset_modem()                       # lte.sh restart\nsubprocess.run(['reboot'], check=False)      # or full device power cycle, then retry\nclient = AtClient()  # fresh client reconnects cleanly\nclient.open_isdr()","handlingStrategy":"retry","validationCode":"from openpilot.common.esim.lpa import AtClient\n\n# ensure modem is responsive at all before attempting ISD-R\nc = AtClient()\nif not c.query('AT'):\n  raise SystemExit('modem not answering basic AT; power-cycle first')","typeGuard":null,"tryCatchPattern":"try:\n  client.open_isdr()\nexcept (RuntimeError, TimeoutError):\n  client._reset_modem()\n  client = AtClient()  # fresh connection after reset\n  client.open_isdr()  # one full retry; escalate to power-cycle if this fails","preventionTips":["Power-cycle the device when lte.sh restart doesn't recover the modem","Confirm eUICC presence before opening ISD-R","Run with DEBUG=1 when failures cluster to capture the underlying AT errors"],"tags":["esim","modem","hardware","retries","serial","python"],"backgroundTag":null,"analyzedSha":"516ec1e68203439a73f340f1d0b3b91eabc626ee","analyzedAt":"2026-08-15T00:17:37.461Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}