{"record":{"id":"576ff7a3c84353cd","repo":"fluent/fluentd","slug":"failed-to-flush-the-buffer-with-secondary-output","errorCode":null,"errorMessage":"failed to flush the buffer with secondary output.","messagePattern":"failed to flush the buffer with secondary output\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"lib/fluent/plugin/output.rb","lineNumber":1387,"sourceCode":"          # @retry.step is called almost as many times as the number of flush threads in a short time.\n          if Time.now >= @retry.next_time\n            @retry.step\n          else\n            @retry.recalc_next_time # to prevent all flush threads from retrying at the same time\n          end\n\n          if @retry.limit?\n            handle_limit_reached(error)\n          elsif error\n            log_retry_error(error, chunk_id_hex, using_secondary)\n          end\n        end\n      end\n\n      def log_retry_error(error, chunk_id_hex, using_secondary)\n        return unless error\n        if using_secondary\n          msg = \"failed to flush the buffer with secondary output.\"\n        else\n          msg = \"failed to flush the buffer.\"\n        end\n        log.warn(msg, retry_times: @retry.steps, next_retry_time: @retry.next_time.round, chunk: chunk_id_hex, error: error)\n        log.warn_backtrace(error.backtrace)\n      end\n\n      def handle_limit_reached(error)\n        if error\n          records = @buffer.queued_records\n          msg = \"Hit limit for retries. dropping all chunks in the buffer queue.\"\n          log.error msg, retry_times: @retry.steps, records: records, error: error\n          log.error_backtrace error.backtrace\n        end\n        @buffer.clear_queue!\n        log.debug \"buffer queue cleared\"\n        @retry = nil\n      end","sourceCodeStart":1369,"sourceCodeEnd":1405,"githubUrl":"https://github.com/fluent/fluentd/blob/dd45c6e18dc7be33b5e5a0f0767bf46307ff5626/lib/fluent/plugin/output.rb#L1369-L1405","documentation":"This is not a raised exception but a log.warn emitted by BufferedOutput#log_retry_error (lib/fluent/plugin/output.rb:1387) via update_retry_state: a buffered chunk failed to flush, retries remain, AND the attempt already fell through to the <secondary> output. Seeing it means the primary destination has been failing long enough to exhaust the primary retry budget and secondary writes are now also erroring — data is still queued and will keep retrying, but the delivery path is degraded on both tiers.","triggerScenarios":"An output with a <secondary> block (e.g. primary @type forward to a collector, secondary @type file to local spool) where the primary exceeded its retry_timeout and switched to secondary, then the secondary write raised too; update_retry_state is called with using_secondary=true and a non-nil error, logging retry_times, next_retry_time, chunk id and the underlying error.","commonSituations":"Remote collector down AND local secondary disk is full/read-only; S3 primary failing on credentials while the secondary archive target also rejects writes; network partition plus permission drift on the fallback path; retry_timeout/secondary_threshold tuned so low that every hiccup skips to secondary.","solutions":["Read the attached 'error' field in the same log line — it identifies the secondary's actual failure (disk full, auth, network)","Restore the secondary first (it is the safety net): free disk space, fix credentials/permissions on the secondary target","Then fix the primary destination and let retries drain the queue naturally; verify chunks decrease with fluentd's buffer metrics / log","Tune <buffer> retry_timeout and secondary_threshold if secondary activation is premature, and check retry_limit so handle_limit_reached (drop-all) never fires","Monitor buffer queued records/total_size and alert before the queue approaches total_limit_size to avoid eventual chunk loss"],"exampleFix":"# before\n<match app.**>\n  @type forward\n  <buffer>\n    retry_timeout 30s\n  </buffer>\n  <secondary>\n    @type file\n    path /var/log/fluent/backup   # disk full -> secondary also fails\n  </secondary>\n</match>\n\n# after\n<match app.**>\n  @type forward\n  <buffer>\n    retry_timeout 72h\n    retry_limit 30\n  </buffer>\n  <secondary>\n    @type file\n    path /mnt/bigvolume/fluent/backup   # writable volume with capacity alerts\n  </secondary>\n</match>","handlingStrategy":"retry","validationCode":"# monitoring pre-check (not error prevention, but early warning):\n# fluentd-prometheus metrics: fluentd_output_status_buffer_total_bytes,\n# fluentd_output_status_retry_count, fluentd_output_status_buffer_queue_length\n# alert when queued bytes > 60% of total_limit_size or retry_count > 0 for 10m","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Provision the secondary with at least as much reliability/capacity as the primary (writable volume, valid credentials, capacity alerts)","Set retry_timeout and retry_limit generously so chunks survive extended outages instead of reaching the drop-all path","Monitor buffer queue depth and retry counters; alert on the first 'failed to flush the buffer' warn, not just the secondary one","Periodically fail over in staging to prove the secondary path actually accepts writes"],"tags":["fluentd","buffer","retry","secondary-output","logging","runtime"],"backgroundTag":"buffer-flush-retry-exhausted","analyzedSha":"dd45c6e18dc7be33b5e5a0f0767bf46307ff5626","analyzedAt":"2026-08-21T16:22:07.332Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}