crowdsecurity/crowdsec · error

while reading file: %w

Error message

while reading file: %w

What it means

downloadDataSet decodes the item's data file as a stream of JSON objects. If json.Decoder.Decode returns an error other than io.EOF, it is wrapped as 'while reading file: %w', meaning the file contains invalid or truncated JSON.

Source

Thrown at pkg/hubops/download.go:114

type DataSet struct {
	Data []enrichment.DataProvider `yaml:"data,omitempty"`
}

// downloadDataSet downloads all the data files for an item.
func downloadDataSet(ctx context.Context, dataFolder string, force bool, reader io.Reader) (bool, error) {
	needReload := false

	dec := yaml.NewDecoder(reader)

	for {
		data := &DataSet{}

		if err := dec.Decode(data); err != nil {
			if errors.Is(err, io.EOF) {
				break
			}

			return needReload, fmt.Errorf("while reading file: %w", err)
		}

		for _, dataS := range data.Data {
			if dataS.SourceURL == "" {
				continue
			}

			// twopenny validation
			if u, err := url.Parse(dataS.SourceURL); err != nil {
				return false, err
			} else if u.Scheme == "" {
				return false, fmt.Errorf("a valid URL was expected (note: local items can download data too): %s", dataS.SourceURL)
			}

			// XXX: check context cancellation
			destPath, err := cwhub.SafePath(dataFolder, dataS.DestPath)
			if err != nil {
				return needReload, err

View on GitHub (pinned to 909b515798)

Solutions

1. Delete the corrupted data file and force a re-download (force=true / 'cscli hub update --force')
2. Verify the data source still emits the expected JSON format
3. Check disk integrity/space if truncation recurs
4. Inspect the wrapped inner error for the exact JSON syntax problem

Example fix

// before
needReload, err := hubops.DownloadDataIfNeeded(ctx, hub, item, false) // reuses bad local file
// after
needReload, err := hubops.DownloadDataIfNeeded(ctx, hub, item, true) // forces fresh download

f, err := os.Open(item.State.LocalPath)
if err != nil { return err }
defer f.Close()
if !json.NewDecoder(f).More() && isTruncated(f) { return fmt.Errorf("data file invalid JSON: %s", item.State.LocalPath) }

_, err := hubops.DownloadDataIfNeeded(ctx, hub, item, true)
if err != nil {
    if strings.Contains(err.Error(), "while reading file") { /* delete file and re-download */ }
    return err
}

When it happens

Trigger: Calling downloadDataSet (via DownloadDataIfNeeded or the data-refresh command) on a data file that is not valid line-delimited JSON — corrupted download, truncated file, or wrong file format.

Common situations: Interrupted download left a partially written data file; upstream source changed format (e.g. plain text instead of JSON); disk corruption.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• failed to parse index: %w
• no appsec_config provided
• no header in message, skipping
• no source user in header message, skipping
• no hub configuration provided

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/f367aebf1b94c868. Report an issue: GitHub.