SigNoz/signoz · warning

agent updater is busy

Error message

agent updater is busy

What it means

UpsertFilterProcessor uses a global atomic lock; if another agent-config update is in flight (lock held), the call returns immediately with 'agent updater is busy'. This is a concurrency guard, not a system failure — only one config mutation runs at a time.

Source

Thrown at pkg/query-service/agentConf/manager.go:288

		}

		opamp.AddToMetricsPipelineSpec("filter")
		configHash, err := opamp.UpsertControlProcessors(ctx, "metrics", processorConf, m.OnConfigUpdate)
		if err != nil {
			slog.ErrorContext(ctx, "failed to call agent config update for trace processor", errors.Attr(err))
			return err
		}

		m.updateDeployStatus(ctx, orgId, opamptypes.ElementTypeSamplingRules, version, opamptypes.DeployInitiated.StringValue(), "Deployment started", configHash, configVersion.Config)
	}

	return nil
}

// UpsertFilterProcessor updates the agent config with new filter processor params
func UpsertFilterProcessor(ctx context.Context, orgId valuer.UUID, version int, config *filterprocessor.Config) error {
	if !atomic.CompareAndSwapUint32(&m.lock, 0, 1) {
		return fmt.Errorf("agent updater is busy")
	}
	defer atomic.StoreUint32(&m.lock, 0)

	// merge current config with new filter params
	// merge current config with new filter params
	processorConf := map[string]interface{}{
		"filter": config,
	}

	opamp.AddToMetricsPipelineSpec("filter")
	configHash, err := opamp.UpsertControlProcessors(ctx, "metrics", processorConf, m.OnConfigUpdate)
	if err != nil {
		slog.ErrorContext(ctx, "failed to call agent config update for trace processor", errors.Attr(err))
		return err
	}

	processorConfYaml, yamlErr := yaml.Marshal(config)
	if yamlErr != nil {

View on GitHub (pinned to 5069bf80b0)

Solutions

1. Retry after the in-flight update completes (short backoff)
2. Serialize config-update calls client-side so only one is outstanding
3. Debounce UI actions that trigger filter upserts

Example fix

// before
err := agentConf.UpsertFilterProcessor(ctx, orgId, version, cfg)
// after
var err error
for i := 0; i < 3; i++ {
    err = agentConf.UpsertFilterProcessor(ctx, orgId, version, cfg)
    if err == nil || !strings.Contains(err.Error(), "busy") {
        break
    }
    time.Sleep(500 * time.Millisecond)
}

Retry with short backoff when the error string contains 'agent updater is busy'; otherwise propagate.

When it happens

Trigger: Two overlapping calls to UpsertFilterProcessor (or other mutations sharing m.lock) from concurrent requests or rapid UI updates.

Common situations: Bulk automation applying filters concurrently, double-submitted UI forms, or retries racing with a slow in-progress update.

Related errors

• CodeTimeout
• failed to update deploy status
• failed to read the stored config correctly
• query %s already started

AI-assisted analysis of SigNoz/signoz@5069bf80b0 (2026-08-28). Data as JSON: /api/errors/6f68112e6799a74b. Report an issue: GitHub.