{"record":{"id":"e3d8036740a4a1aa","repo":"commaai/openpilot","slug":"at-command-timed-out","errorCode":null,"errorMessage":"AT command timed out","messagePattern":"AT command timed out","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"openpilot/common/esim/lpa.py","lineNumber":166,"sourceCode":"          self.query(f\"AT+CCHC={self.channel}\")\n        except (RuntimeError, TimeoutError):\n          pass\n        self.channel = None\n    finally:\n      if self._serial:\n        self._serial.close()\n\n  def _send(self, cmd: str) -> None:\n    if DEBUG:\n      print(f\"SER >> {cmd}\", file=sys.stderr)\n    self._serial.write((cmd + \"\\r\").encode(\"ascii\"))\n\n  def _expect(self) -> list[str]:\n    lines: list[str] = []\n    while True:\n      raw = self._serial.readline()\n      if not raw:\n        raise TimeoutError(\"AT command timed out\")\n      line = raw.decode(errors=\"ignore\").strip()\n      if not line:\n        continue\n      if DEBUG:\n        print(f\"SER << {line}\", file=sys.stderr)\n      if line == \"OK\":\n        return lines\n      if line == \"ERROR\" or line.startswith(\"+CME ERROR\"):\n        raise RuntimeError(f\"AT command failed: {line}\")\n      lines.append(line)\n\n  def _ensure_serial(self, reconnect: bool = False) -> None:\n    if reconnect:\n      self.channel = None\n      try:\n        if self._serial:\n          self._serial.close()\n      except Exception:","sourceCodeStart":148,"sourceCodeEnd":184,"githubUrl":"https://github.com/commaai/openpilot/blob/516ec1e68203439a73f340f1d0b3b91eabc626ee/openpilot/common/esim/lpa.py#L148-L184","documentation":"AtClient._expect() reads lines from the modem's AT serial port until it sees 'Ok'. pyserial's readline() returns empty bytes on timeout; the code interprets an empty read as the modem not answering within the serial timeout and raises TimeoutError('AT command timed out').","triggerScenarios":"Any AtClient.query()/AT exchange where the modem sends nothing within the configured serial read timeout (DEFAULT_TIMEOUT=5s): an unresponsive or crashed modem firmware, a busy modem processing a long operation (e.g. CGLA APDU exchanges), wrong baud rate, or the serial port silently disconnected.","commonSituations":"Modem firmware hang or crash mid-session; the AT port held by another process (no exclusive lock, interleaved reads consume responses); flow control or baud mismatch on custom hardware; modem in a low-power state that stops responding.","solutions":["Retry with backoff — transient modem stalls often recover; open_isdr/send_apdu already wrap this in retry loops, so callers of query() directly should too","Power-cycle or reset the modem (the code path uses _reset_modem() via /usr/comma/lte/lte.sh) and reopen the serial connection","Check no other process is reading /dev/modem_at0 (the code uses /dev/shm/modem.lock — ensure you hold it)","Increase the serial timeout if the modem is legitimately slow (long APDU operations can exceed 5s) and verify baud settings match DEFAULT_BAUD=9600"],"exampleFix":"# before\nlines = client.query('AT+CGLA=...')  # TimeoutError: AT command timed out\n\n# after\nfrom openpilot.common.esim.lpa import AtClient, TimeoutError as AtTimeout\n\nfor attempt in range(3):\n  try:\n    lines = client.query('AT+CGLA=...')\n    break\n  except AtTimeout:\n    client._ensure_serial(reconnect=True)\nelse:\n  raise RuntimeError('modem unresponsive after retries')","handlingStrategy":"retry","validationCode":"import os\n\nif not os.path.exists('/dev/modem_at0'):\n  raise SystemExit('modem AT port missing; modem not ready')","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n  try:\n    lines = client.query('AT+...')\n    break\n  except TimeoutError:\n    client._ensure_serial(reconnect=True)\nelse:\n  client._reset_modem()\n  raise","preventionTips":["Hold /dev/shm/modem.lock so only one process talks to the AT port","Use short commands and re-open the serial link between retry rounds","Reset the modem proactively after any prior AT failure before a new session"],"tags":["serial","modem","timeout","hardware","esim","python"],"backgroundTag":null,"analyzedSha":"516ec1e68203439a73f340f1d0b3b91eabc626ee","analyzedAt":"2026-08-15T00:17:37.461Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}