{"record":{"id":"f367aebf1b94c868","repo":"crowdsecurity/crowdsec","slug":"while-reading-file-w","errorCode":null,"errorMessage":"while reading file: %w","messagePattern":"while reading file: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/hubops/download.go","lineNumber":114,"sourceCode":"type DataSet struct {\n\tData []enrichment.DataProvider `yaml:\"data,omitempty\"`\n}\n\n// downloadDataSet downloads all the data files for an item.\nfunc downloadDataSet(ctx context.Context, dataFolder string, force bool, reader io.Reader) (bool, error) {\n\tneedReload := false\n\n\tdec := yaml.NewDecoder(reader)\n\n\tfor {\n\t\tdata := &DataSet{}\n\n\t\tif err := dec.Decode(data); err != nil {\n\t\t\tif errors.Is(err, io.EOF) {\n\t\t\t\tbreak\n\t\t\t}\n\n\t\t\treturn needReload, fmt.Errorf(\"while reading file: %w\", err)\n\t\t}\n\n\t\tfor _, dataS := range data.Data {\n\t\t\tif dataS.SourceURL == \"\" {\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\t// twopenny validation\n\t\t\tif u, err := url.Parse(dataS.SourceURL); err != nil {\n\t\t\t\treturn false, err\n\t\t\t} else if u.Scheme == \"\" {\n\t\t\t\treturn false, fmt.Errorf(\"a valid URL was expected (note: local items can download data too): %s\", dataS.SourceURL)\n\t\t\t}\n\n\t\t\t// XXX: check context cancellation\n\t\t\tdestPath, err := cwhub.SafePath(dataFolder, dataS.DestPath)\n\t\t\tif err != nil {\n\t\t\t\treturn needReload, err","sourceCodeStart":96,"sourceCodeEnd":132,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/hubops/download.go#L96-L132","documentation":"downloadDataSet decodes the item's data file as a stream of JSON objects. If json.Decoder.Decode returns an error other than io.EOF, it is wrapped as 'while reading file: %w', meaning the file contains invalid or truncated JSON.","triggerScenarios":"Calling downloadDataSet (via DownloadDataIfNeeded or the data-refresh command) on a data file that is not valid line-delimited JSON — corrupted download, truncated file, or wrong file format.","commonSituations":"Interrupted download left a partially written data file; upstream source changed format (e.g. plain text instead of JSON); disk corruption.","solutions":["Delete the corrupted data file and force a re-download (force=true / 'cscli hub update --force')","Verify the data source still emits the expected JSON format","Check disk integrity/space if truncation recurs","Inspect the wrapped inner error for the exact JSON syntax problem"],"exampleFix":"// before\nneedReload, err := hubops.DownloadDataIfNeeded(ctx, hub, item, false) // reuses bad local file\n// after\nneedReload, err := hubops.DownloadDataIfNeeded(ctx, hub, item, true) // forces fresh download","handlingStrategy":"validation","validationCode":"f, err := os.Open(item.State.LocalPath)\nif err != nil { return err }\ndefer f.Close()\nif !json.NewDecoder(f).More() && isTruncated(f) { return fmt.Errorf(\"data file invalid JSON: %s\", item.State.LocalPath) }","typeGuard":null,"tryCatchPattern":"_, err := hubops.DownloadDataIfNeeded(ctx, hub, item, true)\nif err != nil {\n    if strings.Contains(err.Error(), \"while reading file\") { /* delete file and re-download */ }\n    return err\n}","preventionTips":["Force re-download after interrupted hub updates","Validate downloaded data files as JSON before use","Ensure disk space to avoid truncated writes"],"tags":["json","hub","data-file"],"backgroundTag":"json-unmarshal-failed","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}