fluent/fluentd · warning · ArgumentError

#{@key_name} does not exist

Error message

#{@key_name} does not exist

What it means

The parser filter reads the field named by key_name from each record via a record accessor (nested paths like $.payload.data supported). When the accessor returns nil, filter_stream raises ArgumentError '<key_name> does not exist' for that record. The surrounding rescue routes the exception to router.emit_error_event only when emit_invalid_record_to_error is true (the default); with reserve_data true the original record (with empty parse results) is still added to the output stream before the raise, otherwise the record is silently dropped.

Source

Thrown at lib/fluent/plugin/filter_parser.rb:66

    def configure(conf)
      compat_parameters_convert(conf, :parser)

      super

      @accessor = record_accessor_create(@key_name)
      @parser = parser_create
    end

    REPLACE_CHAR = '?'.freeze

    def filter_stream(tag, es)
      new_es = Fluent::MultiEventStream.new
      es.each do |time, record|
        begin
          raw_value = @accessor.call(record)
          if raw_value.nil?
            new_es.add(time, handle_parsed(tag, record, time, {})) if @reserve_data
            raise ArgumentError, "#{@key_name} does not exist"
          else
            filter_one_record(tag, time, record, raw_value) do |result_time, result_record|
              new_es.add(result_time, result_record)
            end
          end
        rescue => e
          router.emit_error_event(tag, time, record, e) if @emit_invalid_record_to_error
        end
      end
      new_es
    end

    private

    def filter_one_record(tag, time, record, raw_value)
      begin
        @parser.parse(raw_value) do |t, values|
          if values

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Correct key_name to match the real field, including record accessor syntax for nested values ($.payload.data).
2. For streams that legitimately lack the field, keep emit_invalid_record_to_error true and route the @ERROR label to a file/sink to inspect misses.
3. Set reserve_data true if records without the field should still flow through (original record kept).
4. Pre-filter with grep to drop or tag records without the key before the parser filter.

Example fix

# before
<filter app.**>
  @type parser
  key_name payload
  <parse>
    @type json
  </parse>
</filter>
# after: misses are kept and routed for inspection
<filter app.**>
  @type parser
  key_name payload
  reserve_data true
  emit_invalid_record_to_error true
  <parse>
    @type json
  </parse>
</filter>
<label @ERROR>
  <match **>
    @type file
    path /var/log/fluent/error/parser-misses
  </match>
</label>

# verify every record has the field before enabling the parser filter
accessor = Fluent::RecordAccessor.new('payload') # or '$.payload.data'
missing = events.count { |_time, record| accessor.call(record).nil? }
abort "#{missing} records lack key_name" if missing > 0

->(record, key) { record.is_a?(Hash) && !record[key].nil? }

# in a custom filter around parser logic:
begin
  filter_stream(tag, es)
rescue ArgumentError => e
  router.emit_error_event(tag, Fluent::EventTime.now, {}, e) if e.message.end_with?('does not exist')
end

When it happens

Trigger: Any event reaching the filter whose record lacks key_name or has it explicitly nil: heterogeneous streams, typo in key_name, wrong nesting (using payload when data lives at $.payload.data), or upstream format changes.

Common situations: Mixed producers where only some events carry the parsed field; key renamed upstream; JSON that sometimes wraps the payload one level deeper; key_name copied from another pipeline.

Related errors

• parse failed #{e.message}
• staged meta file is broken. #{e.message}
• enqueued file chunk is empty
• enqueued meta file is broken. #{e.message}
• Invalid chunk found. unique_id and key not exist: #{@path}

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/84766f85d376e656. Report an issue: GitHub.