What causes "the key specified by 'hostname_key' in cannot be used in buffering chunk key." in fluent/fluentd?

Move hostname injection upstream: use a filter (record_modifier with ${hostname}) before the output, then keep host as a chunk key - injection then happens before chunking, which is what the built-in log error recommends

fluent/fluentd · error · Fluent::ConfigError

the key specified by 'hostname_key' in <inject> cannot be us

Error message

the key specified by 'hostname_key' in <inject> cannot be used in buffering chunk key.

What it means

Fluent::ConfigError from the Inject helper's configure. When hostname_key is set in <inject>, the helper writes the hostname into each record during format processing, which happens after the output plugin has already assigned records to buffer chunks. Chunking on a key that is injected only after chunk assignment would silently misroute records, so configure rejects any buffer chunk key equal to the inject hostname_key. A companion log line ('Use filters to inject hostname to use it in buffer chunking.') states the intended alternative.

Source

Thrown at lib/fluent/plugin_helper/inject.rb:116

        @_inject_tag_key = nil
        @_inject_time_key = nil
        @_inject_time_formatter = nil
      end

      def configure(conf)
        super

        if @inject_config
          @_inject_hostname_key = @inject_config.hostname_key
          if @_inject_hostname_key
            if self.respond_to?(:buffer_config)
              # Output plugin cannot use "hostname"(specified by @hostname_key),
              # injected by this plugin helper, in chunk keys.
              # This plugin helper works in `#format` (in many cases), but modified record
              # don't have any side effect in chunking of output plugin.
              if self.buffer_config.chunk_keys.include?(@_inject_hostname_key)
                log.error "Use filters to inject hostname to use it in buffer chunking."
                raise Fluent::ConfigError, "the key specified by 'hostname_key' in <inject> cannot be used in buffering chunk key."
              end
            end

            @_inject_hostname =  @inject_config.hostname
            unless @_inject_hostname
              @_inject_hostname = ::Socket.gethostname
              log.info "using hostname for specified field", host_key: @_inject_hostname_key, host_name: @_inject_hostname
            end
          end
          @_inject_worker_id_key = @inject_config.worker_id_key
          if @_inject_worker_id_key
            @_inject_worker_id = fluentd_worker_id # get id here, because #with_worker_config method may be used only for #configure in tests
          end
          @_inject_tag_key = @inject_config.tag_key
          @_inject_time_key = @inject_config.time_key
          if @_inject_time_key
            @_inject_time_formatter = case @inject_config.time_type
                                      when :float then ->(time){ time.to_r.truncate(+6).to_f } # microsecond floating point value

View on GitHub (pinned to dd45c6e18d)

Solutions

Move hostname injection upstream: use a filter (record_modifier with ${hostname}) before the output, then keep host as a chunk key - injection then happens before chunking, which is what the built-in log error recommends
Alternatively remove the hostname from the buffer chunk keys and let <inject> hostname_key only enrich the emitted records

Example fix

# before
<match app.**>
  @type file
  <inject>
    hostname_key host
  </inject>
  <buffer>
    keys host
  </buffer>
</match>

# after
<filter app.**>
  @type record_modifier
  <record>
    host ${hostname}
  </record>
</filter>
<match app.**>
  @type file
  <buffer>
    keys host
  </buffer>
</match>

Defensive patterns

Strategy: validation

Validate before calling

# reject the conflict before Fluent's configure does
if inject_config&.hostname_key &&
   respond_to?(:buffer_config) &&
   buffer_config.chunk_keys.include?(inject_config.hostname_key)
  raise Fluent::ConfigError, 'remove hostname_key from chunk keys or inject via a filter'
end

Prevention

Never list an inject-provided key among buffer chunk keys
When chunking on hostname, inject it with a record_modifier filter upstream
Add a config lint rule: chunk keys must be produced before the output plugin

When it happens

Trigger: An output plugin that mixes Inject and Buffer helpers, configured with e.g. <inject>hostname_key host</inject> together with <buffer>keys host</buffer> or <buffer host>. The check buffer_config.chunk_keys.include?(@_inject_hostname_key) fires during configure.

Common situations: Trying to partition output files or tags by the forwarding node's hostname using inject hostname_key plus a chunk key; migrating a record_modifier-based hostname injection into <inject> while leaving host in the buffer keys.

Related errors

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/a47423b6bbe5919f. Report an issue: GitHub.