instructure/canvas-lms · error · DatabaseReplicationError

Replica failed to catch up in the allotted time

Error message

Replica failed to catch up in the allotted time

What it means

DatabaseReplicationError raised in ReportHelper#compile_parallel_report when AccountReport.wait_for_replication fails both checks: the report replica did not catch up to the source WAL position within 120 seconds and the secondary did not within 30 minutes. The report is only safe to read from a replica once it has replicated to that xlog location.

Solutions

1. Check replica health/lag (pg_stat_replication, replay_lag) and restore the replica connection or restart WAL replay.
2. Increase the wait_for_replication timeouts if lag is routinely large.
3. Re-run the report after replicas catch up; the report_runner has been marked failed.
4. If latency is chronic, provision a faster replica or route the report to the primary explicitly.
5. Check report_runner.fail/fail_with_error output for the underlying replication state at failure time.

// check replica lag before launching the report
lag = ActiveRecord::Base.connection.select_one(
  "SELECT EXTRACT(EPOCH FROM now() - pg_last_xact_replay_timestamp()) AS lag"
)['lag'].to_i
raise 'replica too far behind' if lag > 300

begin
  run_account_report_runner(report, params)
rescue DatabaseReplicationError => e
  report.update( last_error: e.message )
  schedule_retry_after_replica_recovery(report)
end

When it happens

Trigger: Running an account report whose xlog_location cannot be replicated within the timeouts — typically a badly lagging or stalled streaming replica, or replica downtime, during compile_parallel_report (invoked via run_account_report_runner).

Common situations: Long-running reports generating huge WAL volume faster than the replica applies it; replica restarted or network-partitioned during a report; overloaded secondary delaying WAL replay.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• operation timed out
• could not retrieve configuration, the server response timed…
• Course with canvas id
• Database error ( )
• Error hiding assignment grades for sections

AI-assisted analysis of instructure/canvas-lms@1c9f0bb801 (2026-09-15). Data as JSON: /api/errors/39e12212afb83be7. Report an issue: GitHub.

Appendix: source

Thrown at gems/plugins/account_reports/lib/account_reports/report_helper.rb:475

      report_runner.fail
      fail_with_error(e)
    ensure
      update_parallel_progress(account_report: @account_report, report_runner:)
      compile_parallel_report(report_runner, headers, files:) if last_account_report_runner?(@account_report)
    end
  end

  def compile_parallel_report(report_runner, headers, files: nil)
    GuardRail.activate(:primary) { @account_report.update(total_lines: @account_report.account_report_rows.count + 1) }
    xlog_location = AccountReport.current_xlog_location
    # wait 2 minutes for report db to catch up, if it does not catch up, use the
    # secondary db when it is caught up.
    replica = if AccountReport.wait_for_replication(start: xlog_location, timeout: 120, use_report: true)
                :report
              elsif AccountReport.wait_for_replication(start: xlog_location, timeout: 30.minutes)
                :secondary
              else
                raise DatabaseReplicationError, "Replica failed to catch up in the allotted time"
              end
    files ? compile_parallel_zip_report(files, replica:) : write_report_from_rows(headers, replica:)
  rescue => e
    report_runner.fail
    fail_with_error(e)
  ensure
    GuardRail.activate(:primary) { @account_report.delete_account_report_rows }
  end

  def write_report_from_rows(headers, replica: :report)
    activate_report_db(replica:) do
      write_report(headers, enable_i18n_features: false, replica:) do |csv|
        @account_report.account_report_rows.order(:account_report_runner_id, :row_number)
                       .find_in_batches(strategy: :cursor) do |batch|
          batch.each { |record| csv << record.row }
        end
      end
    end

View on GitHub (pinned to 1c9f0bb801)