fluent/fluentd · error · Fluent::ConfigError

regexp#{i} contains a duplicated key, #{key}

Error message

regexp#{i} contains a duplicated key, #{key}

What it means

While parsing the deprecated numbered grep directives, configure builds a hash keyed by record field name. If two regexpN lines reference the same field key, the second is rejected with Fluent::ConfigError because each key may only carry one AND-condition pattern in this syntax.

Source

Thrown at lib/fluent/plugin/filter_grep.rb:100

        config_param :key, :string
        desc "The regular expression."
        config_param :pattern, :regexp
      end
    end

    def configure(conf)
      super

      regexp_and_conditions = {}
      regexp_or_conditions = {}
      exclude_and_conditions = {}
      exclude_or_conditions = {}

      (1..REGEXP_MAX_NUM).each do |i|
        next unless conf["regexp#{i}"]
        key, regexp = conf["regexp#{i}"].split(/ /, 2)
        raise Fluent::ConfigError, "regexp#{i} does not contain 2 parameters" unless regexp
        raise Fluent::ConfigError, "regexp#{i} contains a duplicated key, #{key}" if regexp_and_conditions[key]
        regexp_and_conditions[key] = Expression.new(record_accessor_create(key), Regexp.compile(regexp))
      end

      (1..REGEXP_MAX_NUM).each do |i|
        next unless conf["exclude#{i}"]
        key, exclude = conf["exclude#{i}"].split(/ /, 2)
        raise Fluent::ConfigError, "exclude#{i} does not contain 2 parameters" unless exclude
        raise Fluent::ConfigError, "exclude#{i} contains a duplicated key, #{key}" if exclude_or_conditions[key]
        exclude_or_conditions[key] = Expression.new(record_accessor_create(key), Regexp.compile(exclude))
      end

      if @regexps.size > 1
        log.info "Top level multiple <regexp> is interpreted as 'and' condition"
      end
      @regexps.each do |e|
        raise Fluent::ConfigError, "Duplicate key: #{e.key}" if regexp_and_conditions.key?(e.key)
        regexp_and_conditions[e.key] = Expression.new(record_accessor_create(e.key), e.pattern)
      end

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Combine the two patterns into one regex: regexp1 message /(a|b)/
2. Use different field keys if the conditions really apply to different fields
3. Migrate to <or> sections for OR semantics over the same key
4. Validate with fluentd --dry-run

Example fix

# before
regexp1 message /error/
regexp2 message /warning/

# after
<or>
  <regexp>
    key message
    pattern /error/
  </regexp>
  <regexp>
    key message
    pattern /warning/
  </regexp>
</or>

keys = (1..20).filter_map { |i| conf["regexp#{i}"]&.split(/ /, 2)&.first }
dupes = keys.tally.select { |_, n| n > 1 }.keys
abort "duplicate regexpN keys: #{dupes.join(', ')}" unless dupes.empty?

begin
  driver = Fluent::Test::Driver::Filter.new(Fluent::Plugin::GrepFilter).configure(conf)
rescue Fluent::ConfigError => e
  # e.message contains the duplicated key; merge into one alternation regex and reconfigure
  raise
end

When it happens

Trigger: Two lines like regexp1 message /a/ and regexp2 message /b/ — same key 'message' parsed from different regexpN indices; also a single line duplicated by a config generator or copy-paste.

Common situations: Trying to express 'field matches X OR Y' with two regexpN lines (this syntax is AND, and duplicate keys are disallowed — the fix is one combined regex or <or> sections); templated configs that stamp the same key repeatedly.

Related errors

• exclude#{i} contains a duplicated key, #{key}
• Duplicate key: #{e.key}
• regexp#{i} does not contain 2 parameters
• exclude#{i} does not contain 2 parameters
• Do not specify both <regexp> and <exclude> in <and>

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/d82fda70144ca587. Report an issue: GitHub.