{"record":{"id":"39e12212afb83be7","repo":"instructure/canvas-lms","slug":"replica-failed-to-catch-up-in-the-allotted-time","errorCode":null,"errorMessage":"Replica failed to catch up in the allotted time","messagePattern":"Replica failed to catch up in the allotted time","errorType":"exception","errorClass":"DatabaseReplicationError","httpStatus":null,"severity":"error","filePath":"gems/plugins/account_reports/lib/account_reports/report_helper.rb","lineNumber":475,"sourceCode":"      report_runner.fail\n      fail_with_error(e)\n    ensure\n      update_parallel_progress(account_report: @account_report, report_runner:)\n      compile_parallel_report(report_runner, headers, files:) if last_account_report_runner?(@account_report)\n    end\n  end\n\n  def compile_parallel_report(report_runner, headers, files: nil)\n    GuardRail.activate(:primary) { @account_report.update(total_lines: @account_report.account_report_rows.count + 1) }\n    xlog_location = AccountReport.current_xlog_location\n    # wait 2 minutes for report db to catch up, if it does not catch up, use the\n    # secondary db when it is caught up.\n    replica = if AccountReport.wait_for_replication(start: xlog_location, timeout: 120, use_report: true)\n                :report\n              elsif AccountReport.wait_for_replication(start: xlog_location, timeout: 30.minutes)\n                :secondary\n              else\n                raise DatabaseReplicationError, \"Replica failed to catch up in the allotted time\"\n              end\n    files ? compile_parallel_zip_report(files, replica:) : write_report_from_rows(headers, replica:)\n  rescue => e\n    report_runner.fail\n    fail_with_error(e)\n  ensure\n    GuardRail.activate(:primary) { @account_report.delete_account_report_rows }\n  end\n\n  def write_report_from_rows(headers, replica: :report)\n    activate_report_db(replica:) do\n      write_report(headers, enable_i18n_features: false, replica:) do |csv|\n        @account_report.account_report_rows.order(:account_report_runner_id, :row_number)\n                       .find_in_batches(strategy: :cursor) do |batch|\n          batch.each { |record| csv << record.row }\n        end\n      end\n    end","sourceCodeStart":457,"sourceCodeEnd":493,"githubUrl":"https://github.com/instructure/canvas-lms/blob/1c9f0bb8013ed69c4f2efe11fd483025469b7e6c/gems/plugins/account_reports/lib/account_reports/report_helper.rb#L457-L493","documentation":"DatabaseReplicationError raised in ReportHelper#compile_parallel_report when AccountReport.wait_for_replication fails both checks: the report replica did not catch up to the source WAL position within 120 seconds and the secondary did not within 30 minutes. The report is only safe to read from a replica once it has replicated to that xlog location.","triggerScenarios":"Running an account report whose xlog_location cannot be replicated within the timeouts — typically a badly lagging or stalled streaming replica, or replica downtime, during compile_parallel_report (invoked via run_account_report_runner).","commonSituations":"Long-running reports generating huge WAL volume faster than the replica applies it; replica restarted or network-partitioned during a report; overloaded secondary delaying WAL replay.","solutions":["Check replica health/lag (pg_stat_replication, replay_lag) and restore the replica connection or restart WAL replay.","Increase the wait_for_replication timeouts if lag is routinely large.","Re-run the report after replicas catch up; the report_runner has been marked failed.","If latency is chronic, provision a faster replica or route the report to the primary explicitly.","Check report_runner.fail/fail_with_error output for the underlying replication state at failure time."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// check replica lag before launching the report\nlag = ActiveRecord::Base.connection.select_one(\n  \"SELECT EXTRACT(EPOCH FROM now() - pg_last_xact_replay_timestamp()) AS lag\"\n)['lag'].to_i\nraise 'replica too far behind' if lag > 300","typeGuard":null,"tryCatchPattern":"begin\n  run_account_report_runner(report, params)\nrescue DatabaseReplicationError => e\n  report.update( last_error: e.message )\n  schedule_retry_after_replica_recovery(report)\nend","preventionTips":["Monitor replication lag and alert before it exceeds the 120s/30min budgets.","Avoid launching huge reports while a replica is down or replaying catch-up.","Set alerting on pg_stat_replication replay_lag.","Document an ops runbook for resuming failed reports after replica recovery."],"tags":["database","replication","timeout"],"backgroundTag":"request-timeout","analyzedSha":"1c9f0bb8013ed69c4f2efe11fd483025469b7e6c","analyzedAt":"2026-09-15T20:33:18.891Z","contentChangedAt":"2026-09-15T20:33:18.891Z","schemaVersion":2},"datasetVersion":"2026-09-23T02:17:17.105Z"}