fluent/fluentd · warning · Fluent::Plugin::Parser::ParserError

parse failed #{e.message}

Error message

parse failed #{e.message}

What it means

Inside filter_one_record, after the configured parser runs: Fluent::Plugin::Parser::ParserError is re-raised unchanged; ArgumentError messages starting with 'invalid byte sequence in' are scrubbed and retried when replace_invalid_sequence is true; every other exception (TypeError from a non-string raw_value, engine-specific errors) is wrapped as ParserError 'parse failed <original message>'. filter_stream's rescue then sends it to emit_error_event when emit_invalid_record_to_error is true (default).

Source

Thrown at lib/fluent/plugin/filter_parser.rb:109

          else
            router.emit_error_event(tag, time, record, Fluent::Plugin::Parser::ParserError.new("pattern not matched with data '#{raw_value}'")) if @emit_invalid_record_to_error
            next unless @reserve_data
            t = time
            values = {}
          end
          yield(t, handle_parsed(tag, record, t, values))
        end

      rescue Fluent::Plugin::Parser::ParserError => e
        raise e
      rescue ArgumentError => e
        raise unless @replace_invalid_sequence
        raise unless e.message.index("invalid byte sequence in") == 0

        raw_value = raw_value.scrub(REPLACE_CHAR)
        retry
      rescue => e
        raise Fluent::Plugin::Parser::ParserError, "parse failed #{e.message}"
      end
    end

    def handle_parsed(tag, record, t, values)
      if values && @inject_key_prefix
        values = Hash[values.map { |k, v| [@inject_key_prefix + k, v] }]
      end
      r = @hash_value_field ? {@hash_value_field => values} : values
      if @reserve_data
        r = r ? record.merge(r) : record
      end
      r
    end
  end
end

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Verify the data actually matches the chosen <parse> type; test a sample record with the same parser settings.
2. If key_name can hold non-strings, pre-route or convert those events (grep filter or a record_transformer) before the parser filter.
3. Set replace_invalid_sequence true to survive encoding issues instead of failing.
4. Keep emit_invalid_record_to_error true and capture @ERROR output to quantify bad records.

Example fix

# before
<filter app.**>
  @type parser
  key_name message
  <parse>
    @type apache2
  </parse>
</filter>
# after: parser matches actual payload shape, bad records land in @ERROR
<filter app.**>
  @type parser
  key_name payload
  emit_invalid_record_to_error true
  replace_invalid_sequence true
  <parse>
    @type json
  </parse>
</filter>

# smoke-test the parser against representative payloads before deploy
parser = Fluent::Plugin.new_parser('json')
parser.configure(parser_config)
parser.parse(sample_payload) { |t, r| abort 'parser failed' unless r }

->(v) { v.is_a?(String) } # key_name must hold a string for line-oriented parsers

begin
  parser.parse(raw_value) { |t, r| yield(t, r) }
rescue Fluent::Plugin::Parser::ParserError => e
  router.emit_error_event(tag, time, record, e) # 'parse failed ...' goes to @ERROR
end

When it happens

Trigger: The parser plugin raises something other than a normal mismatch while processing raw_value: key_name holds a non-String value (Array/Hash) fed to a line-based parser, parser-specific runtime errors, or malformed input that is not an 'invalid byte sequence' ArgumentError.

Common situations: JSON fields that sometimes contain arrays/objects instead of a string to parse; parsers with format assumptions (apache2, syslog) receiving foreign data; binary/garbage payloads from misconfigured senders.

Related errors

• #{@key_name} does not exist
• staged meta file is broken. #{e.message}
• enqueued file chunk is empty
• enqueued meta file is broken. #{e.message}
• Invalid chunk found. unique_id and key not exist: #{@path}

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/6cfa4f9196bcbb29. Report an issue: GitHub.