apache/druid · warning

Counter increment amount must be non-negative, but got value

Error message

Counter increment amount must be non-negative, but got value[%f] for metric[%s]. If this is valid, metric should be defined as a gauge instead of counter.

What it means

Prometheus counters can only increase; the client library rejects negative inc(). When a Druid metric arrives with a negative value while the metric is defined as a Counter in the emitter's configured metrics list, the increment is skipped and this warning explains that the metric should be a Gauge if negative values are valid.

Solutions

1. Change the metric's type from 'counter' to 'gauge' in the prometheus emitter's metrics configuration if negative values are legitimate
2. Check the emitting component for a bug producing negative values where only non-negative makes sense
3. Verify which metric name the warning names and confirm its Druid semantics before reclassifying
4. If the value must be a counter, clamp or track deltas instead of absolute values

Example fix

// before (emitter JSON config)
{"name": "ingest/events/lag", "type": "counter"}
// after
{"name": "ingest/events/lag", "type": "gauge"}

if (value < 0 && isCounterMetric(metricName)) {
  log.warn("Skipping negative value %f for counter %s", value, metricName);
  return;
}

When it happens

Trigger: Emitting a Druid metric whose value is negative while prometheus-emitter config maps that metric name to type 'counter' — e.g. lag or delta metrics that legitimately go negative, or buggy metric producers sending negative deltas.

Common situations: Kafka/Rabbit lag metrics temporarily negative due to clock/offset quirks; metrics like query/time deltas or queue depths misconfigured as counters; copying a metric config line as 'counter' when it should be 'gauge'.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• count is negative,
• Cpu time must enabled
• Dimension name cannot be null
• Emissions of events not successful
• Encountered metric with null or empty name at position

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/2dc01583a3818608. Report an issue: GitHub.

Appendix: source

Thrown at extensions-contrib/prometheus-emitter/src/main/java/org/apache/druid/emitter/prometheus/PrometheusEmitter.java:187

          if (config.isAddHostAsLabel() && TAG_HOSTNAME.equals(labelName)) {
            labelValues[i] = host;
          } else if (config.isAddServiceAsLabel() && TAG_SERVICE.equals(labelName)) {
            labelValues[i] = service;
          } else if (extraLabels.containsKey(labelName)) {
            labelValues[i] = config.getExtraLabels().get(labelName);
          } else {
            labelValues[i] = "unknown";
          }
        }
      }

      if (metric.getCollector() instanceof Counter) {
        double metricValue = value.doubleValue();
        if (metricValue >= 0) {
          ((Counter) metric.getCollector()).labels(labelValues).inc(metricValue);
          metric.resetLastUpdateTime(Arrays.asList(labelValues));
        } else {
          log.warn("Counter increment amount must be non-negative, but got value[%f] for metric[%s]. If this is valid, metric should be defined as a gauge instead of counter.", metricValue, name);
        }
      } else if (metric.getCollector() instanceof Gauge) {
        ((Gauge) metric.getCollector()).labels(labelValues).set(value.doubleValue());
        metric.resetLastUpdateTime(Arrays.asList(labelValues));
      } else if (metric.getCollector() instanceof Histogram) {
        ((Histogram) metric.getCollector()).labels(labelValues)
                                           .observe(value.doubleValue() / metric.getConversionFactor());
        metric.resetLastUpdateTime(Arrays.asList(labelValues));
      } else {
        log.error("Unrecognized metric type [%s]", metric.getCollector().getClass());
      }
    } else {
      log.debug("Unmapped metric [%s]", name);
    }
  }

  private void pushMetric()
  {

View on GitHub (pinned to 9b90983fd2)