What causes "unable to insert bulk" in crowdsecurity/crowdsec?

Read the wrapped inner error for the true cause (constraint, connection, packet size).

unable to insert bulk

Error message

unable to insert bulk

What it means

BulkError is a sentinel error returned when persisting an alert (and its events/decisions) into the database fails: the alert builder Save, the transaction creation, or a bulk insert fails. It wraps the underlying ent/SQL error so errors.Is matching still works.

Source

Thrown at pkg/database/errors.go:17

package database

import "errors"

var (
	UserExists        = errors.New("user already exist")
	UserNotExists     = errors.New("user doesn't exist")
	HashError         = errors.New("unable to hash")
	InsertFail        = errors.New("unable to insert row")
	QueryFail         = errors.New("unable to query")
	UpdateFail        = errors.New("unable to update")
	DeleteFail        = errors.New("unable to delete")
	ItemNotFound      = errors.New("object not found")
	ParseTimeFail     = errors.New("unable to parse time")
	ParseDurationFail = errors.New("unable to parse duration")
	MarshalFail       = errors.New("unable to serialize")
	BulkError         = errors.New("unable to insert bulk")
	ParseType         = errors.New("unable to parse type")
	InvalidIPOrRange  = errors.New("invalid ip address / range")
	InvalidFilter     = errors.New("invalid filter")
)

View on GitHub (pinned to 909b515798)

Solutions

Read the wrapped inner error for the true cause (constraint, connection, packet size).
Retry on transient connection errors; crowdsec's DB driver may need higher timeouts in db_config.
Reduce batch size (fewer alerts/events per call) to stay under DB packet limits.
Check disk space and DB health; verify schema is migrated.

Example fix

// before
for _, alert := range thousandsOfAlerts {
    client.CreateAlert(ctx, alert) // one giant bulk insert fails
}
// after
for chunk := range slices.Chunk(thousandsOfAlerts, 50) {
    if _, err := client.CreateAlerts(ctx, chunk); err != nil {
        return fmt.Errorf("batch: %w", err)
    }
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Go: keep batches bounded and DB reachable before bulk writes
if len(alerts) > 100 { alerts = alerts[:100] }
if err := db.PingContext(ctx); err != nil { return err }

Type guard

func isBulkError(err error) bool { return errors.Is(err, database.BulkError) }

Try / catch

if err := saveAlerts(ctx, alerts); err != nil {
    if errors.Is(err, database.BulkError) {
        log.Warnf("bulk save failed: %v", err)
        return retryWithBackoff(alerts)
    }
    return err
}

Prevention

Batch inserts in modest sizes (avoid max_allowed_packet limits)
Add retry with backoff for transient connection errors
Monitor DB disk space and connection health

When it happens

Trigger: alertB.Save(ctx) failing on constraint violations; c.Ent.Tx(ctx) failing to open a transaction; createAlertBatch/saveAlerts bulk-inserting events or decisions and hitting an error; UpdateCommunityBlocklist bulk upserts failing.

Common situations: DB connection dropped mid-insert (MySQL 'server has gone away'), unique constraint collisions on concurrent alert ingestion, disk-full database, oversized batch exceeding max_allowed_packet on MySQL.

Related errors

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/0552c97fc45c4713. Report an issue: GitHub.