fluent/fluentd · error · Fluent::ConfigError

exclude#{i} contains a duplicated key, #{key}

Error message

exclude#{i} contains a duplicated key, #{key}

What it means

While parsing deprecated excludeN directives, configure tracks fields already used by exclude conditions (exclude_or_conditions hash). A second excludeN line naming the same key raises Fluent::ConfigError because the key is duplicated in the OR-condition set.

Source

Thrown at lib/fluent/plugin/filter_grep.rb:108

      regexp_and_conditions = {}
      regexp_or_conditions = {}
      exclude_and_conditions = {}
      exclude_or_conditions = {}

      (1..REGEXP_MAX_NUM).each do |i|
        next unless conf["regexp#{i}"]
        key, regexp = conf["regexp#{i}"].split(/ /, 2)
        raise Fluent::ConfigError, "regexp#{i} does not contain 2 parameters" unless regexp
        raise Fluent::ConfigError, "regexp#{i} contains a duplicated key, #{key}" if regexp_and_conditions[key]
        regexp_and_conditions[key] = Expression.new(record_accessor_create(key), Regexp.compile(regexp))
      end

      (1..REGEXP_MAX_NUM).each do |i|
        next unless conf["exclude#{i}"]
        key, exclude = conf["exclude#{i}"].split(/ /, 2)
        raise Fluent::ConfigError, "exclude#{i} does not contain 2 parameters" unless exclude
        raise Fluent::ConfigError, "exclude#{i} contains a duplicated key, #{key}" if exclude_or_conditions[key]
        exclude_or_conditions[key] = Expression.new(record_accessor_create(key), Regexp.compile(exclude))
      end

      if @regexps.size > 1
        log.info "Top level multiple <regexp> is interpreted as 'and' condition"
      end
      @regexps.each do |e|
        raise Fluent::ConfigError, "Duplicate key: #{e.key}" if regexp_and_conditions.key?(e.key)
        regexp_and_conditions[e.key] = Expression.new(record_accessor_create(e.key), e.pattern)
      end

      if @excludes.size > 1
        log.info "Top level multiple <exclude> is interpreted as 'or' condition"
      end
      @excludes.each do |e|
        raise Fluent::ConfigError, "Duplicate key: #{e.key}" if exclude_or_conditions.key?(e.key)
        exclude_or_conditions[e.key] = Expression.new(record_accessor_create(e.key), e.pattern)
      end

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Merge into one regex: exclude1 level /(debug|trace)/
2. Use distinct keys if the conditions target different fields
3. Migrate to multiple <exclude> sections, which are already interpreted as OR conditions
4. Validate with fluentd --dry-run

Example fix

# before
exclude1 level debug
exclude2 level trace

# after
<exclude>
  key level
  pattern /debug|trace/
</exclude>

keys = (1..20).filter_map { |i| conf["exclude#{i}"]&.split(/ /, 2)&.first }
dupes = keys.tally.select { |_, n| n > 1 }.keys
abort "duplicate excludeN keys: #{dupes.join(', ')}" unless dupes.empty?

begin
  Fluent::Plugin::GrepFilter.new.configure(conf)
rescue Fluent::ConfigError => e
  # message contains the duplicated key; merge into /(x|y)/ and retry configure
  raise
end

When it happens

Trigger: Two lines such as exclude1 level debug and exclude2 level trace — the same key appears under two excludeN indices; duplicated generated lines in templated configs.

Common situations: Attempting 'exclude if field is X or Y' with two excludeN lines (the correct form is one combined regex or multiple <exclude> sections which are OR'ed); copy-paste of an exclude line with only the pattern changed but not the key.

Related errors

• regexp#{i} contains a duplicated key, #{key}
• Duplicate key: #{e.key}
• regexp#{i} does not contain 2 parameters
• exclude#{i} does not contain 2 parameters
• Do not specify both <regexp> and <exclude> in <and>

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/b8be441216d9c767. Report an issue: GitHub.