What causes "{func.__name__} failed after retry" in commaai/openpilot?

Fix the underlying failure: read the printed ' failed, trying again' messages and the original traceback context

commaai/openpilot · error · Exception

{func.name} failed after retry

Error message

{func.__name__} failed after retry

What it means

Raised by the retry decorator in common/utils.py after the wrapped function raised on every one of `attempts` iterations. Each failure is printed and the loop sleeps `delay` seconds; when attempts are exhausted the generic Exception is raised unless ignore_failure=True, in which case None is returned (implicitly).

Source

Thrown at openpilot/common/utils.py:274

    lines.append(gap.join(_align(row[i], widths[i]) for i in range(ncols)))
  return "\n".join(lines)


def retry(attempts=3, delay=1.0, ignore_failure=False):
  def decorator(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
      for _ in range(attempts):
        try:
          return func(*args, **kwargs)
        except Exception:
          print(f"{func.__name__} failed, trying again")
          time.sleep(delay)

      if ignore_failure:
        print(f"{func.__name__} failed after retry")
      else:
        raise Exception(f"{func.__name__} failed after retry")
    return wrapper
  return decorator

View on GitHub (pinned to 516ec1e682)

Solutions

Fix the underlying failure: read the printed '<fn> failed, trying again' messages and the original traceback context
Increase attempts/delay only if the failure is genuinely transient (network flaps)
Set ignore_failure=True (or use the decorator's argument) if the call is best-effort
Note the original exception is lost; reproduce locally without the decorator to get the real traceback

Example fix

// before
@retry(attempts=5, delay=1)
def fetch(): ...

// after
def fetch(): ...
for i in range(5):
  try:
    return fetch()
  except Exception as e:
    if i == 4:
      raise

Defensive patterns

Strategy: retry

Try / catch

try:
    result = decorated_fn()
except Exception as e:
    if str(e).endswith('failed after retry'):
        # original exception was swallowed; reproduce it without the decorator
        raise RuntimeError('underlying call keeps failing; run fn directly for traceback') from e
    raise

Prevention

Keep retryable functions small and side-effect free so retries are safe
Log the underlying exception, not just the function name, when wrapping with @retry
Size attempts/delay to the real recovery time of the dependency (e.g. network reconnect)

When it happens

Trigger: A decorated function (e.g., a network fetch or params write with retry(decorator)) raising consistently: bad URL, missing dependency, permission error. Every call raises, attempts run out, and the original exception is discarded in favor of the generic one.

Common situations: Network endpoints down or DNS failing so all N attempts fail, a bug in the wrapped function so it always raises, or CI environments without connectivity. The bare 'print' + swallowed exception makes root-cause hunting hard.

Related errors

AI-assisted analysis of commaai/openpilot@516ec1e682 (2026-08-15). Data as JSON: /api/errors/9ece4f9f816a7b85. Report an issue: GitHub.