commaai/openpilot · error · TimeoutError

AT command timed out

Error message

AT command timed out

What it means

AtClient._expect() reads lines from the modem's AT serial port until it sees 'Ok'. pyserial's readline() returns empty bytes on timeout; the code interprets an empty read as the modem not answering within the serial timeout and raises TimeoutError('AT command timed out').

Source

Thrown at openpilot/common/esim/lpa.py:166

          self.query(f"AT+CCHC={self.channel}")
        except (RuntimeError, TimeoutError):
          pass
        self.channel = None
    finally:
      if self._serial:
        self._serial.close()

  def _send(self, cmd: str) -> None:
    if DEBUG:
      print(f"SER >> {cmd}", file=sys.stderr)
    self._serial.write((cmd + "\r").encode("ascii"))

  def _expect(self) -> list[str]:
    lines: list[str] = []
    while True:
      raw = self._serial.readline()
      if not raw:
        raise TimeoutError("AT command timed out")
      line = raw.decode(errors="ignore").strip()
      if not line:
        continue
      if DEBUG:
        print(f"SER << {line}", file=sys.stderr)
      if line == "OK":
        return lines
      if line == "ERROR" or line.startswith("+CME ERROR"):
        raise RuntimeError(f"AT command failed: {line}")
      lines.append(line)

  def _ensure_serial(self, reconnect: bool = False) -> None:
    if reconnect:
      self.channel = None
      try:
        if self._serial:
          self._serial.close()
      except Exception:

View on GitHub (pinned to 516ec1e682)

Solutions

1. Retry with backoff — transient modem stalls often recover; open_isdr/send_apdu already wrap this in retry loops, so callers of query() directly should too
2. Power-cycle or reset the modem (the code path uses _reset_modem() via /usr/comma/lte/lte.sh) and reopen the serial connection
3. Check no other process is reading /dev/modem_at0 (the code uses /dev/shm/modem.lock — ensure you hold it)
4. Increase the serial timeout if the modem is legitimately slow (long APDU operations can exceed 5s) and verify baud settings match DEFAULT_BAUD=9600

Example fix

# before
lines = client.query('AT+CGLA=...')  # TimeoutError: AT command timed out

# after
from openpilot.common.esim.lpa import AtClient, TimeoutError as AtTimeout

for attempt in range(3):
  try:
    lines = client.query('AT+CGLA=...')
    break
  except AtTimeout:
    client._ensure_serial(reconnect=True)
else:
  raise RuntimeError('modem unresponsive after retries')

import os

if not os.path.exists('/dev/modem_at0'):
  raise SystemExit('modem AT port missing; modem not ready')

for attempt in range(3):
  try:
    lines = client.query('AT+...')
    break
  except TimeoutError:
    client._ensure_serial(reconnect=True)
else:
  client._reset_modem()
  raise

When it happens

Trigger: Any AtClient.query()/AT exchange where the modem sends nothing within the configured serial read timeout (DEFAULT_TIMEOUT=5s): an unresponsive or crashed modem firmware, a busy modem processing a long operation (e.g. CGLA APDU exchanges), wrong baud rate, or the serial port silently disconnected.

Common situations: Modem firmware hang or crash mid-session; the AT port held by another process (no exclusive lock, interleaved reads consume responses); flow control or baud mismatch on custom hardware; modem in a low-power state that stops responding.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Failed to open ISD-R after retries
• no eUICC detected
• AT command failed: {line}
• Missing +CGLA response
• no profile at index {ref} (have {len(profiles)})

AI-assisted analysis of commaai/openpilot@516ec1e682 (2026-08-15). Data as JSON: /api/errors/e3d8036740a4a1aa. Report an issue: GitHub.