{"record":{"id":"616023e7e677c7c7","repo":"Tencent/WeKnora","slug":"read-source-file-w","errorCode":null,"errorMessage":"read source file: %w","messagePattern":"read source file: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/application/service/temporary_document.go","lineNumber":358,"sourceCode":"\t\t})\n\t}\n\tchunksJSON, _ := json.Marshal(chunks)\n\timagesJSON, _ := json.Marshal(images)\n\tmetadataJSON, _ := json.Marshal(metadata)\n\treturn s.repo.MarkReady(ctx, payload.TenantID, payload.DocumentID, content,\n\t\ttypes.JSON(chunksJSON), types.JSON(imagesJSON), types.JSON(metadataJSON),\n\t\tchunker.ApproxTokenCount(content, lang), len(chunks), time.Now())\n}\n\nfunc (s *temporaryDocumentService) parse(ctx context.Context, document *types.TemporaryDocument) (string, []types.TemporaryDocumentImage, map[string]string, error) {\n\tfile, err := s.fileService.GetFile(ctx, document.ResourceRef)\n\tif err != nil {\n\t\treturn \"\", nil, nil, fmt.Errorf(\"open source file: %w\", err)\n\t}\n\tdefer file.Close()\n\tdata, err := io.ReadAll(io.LimitReader(file, secutils.GetMaxFileSizeMB()*1024*1024+1))\n\tif err != nil {\n\t\treturn \"\", nil, nil, fmt.Errorf(\"read source file: %w\", err)\n\t}\n\text := document.FileType\n\tvar options types.TemporaryDocumentCreateOptions\n\t_ = json.Unmarshal(document.ProcessingOptions, &options)\n\tif options.ParserEngine == \"\" || options.ParserEngine == \"auto\" {\n\t\tif tenant, ok := ctx.Value(types.TenantInfoContextKey).(*types.Tenant); ok && tenant != nil {\n\t\t\toptions.ParserEngine = tenant.ParserEngineConfig.ResolveChatParserEngine(ext)\n\t\t}\n\t}\n\tif _, ok := temporaryTextExtensions[ext]; ok && (options.ParserEngine == \"\" || options.ParserEngine == \"auto\") {\n\t\treturn string(data), nil, map[string]string{\"parser\": \"plain_text\"}, nil\n\t}\n\tif docparser.IsAudioFormat(ext) {\n\t\tif options.ASRModelID == \"\" {\n\t\t\treturn \"\", nil, nil, fmt.Errorf(\"audio transcription model is not configured\")\n\t\t}\n\t\tasrModel, err := s.modelService.GetASRModel(ctx, options.ASRModelID)\n\t\tif err != nil {","sourceCodeStart":340,"sourceCodeEnd":376,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/application/service/temporary_document.go#L340-L376","documentation":"After opening the source file, parse reads it with io.ReadAll under a LimitReader capped at the max file size (+1 byte to detect overflow); an I/O error during the read is wrapped as 'read source file'. This happens after a successful open, so the problem is mid-stream: connection reset, disk error, or truncated object.","triggerScenarios":"Storage stream breaking mid-read (network drop to S3/GCS); underlying object truncated or corrupted; local disk read errors; timeout killing the connection partway through.","commonSituations":"Large files over flaky network paths; proxy/load-balancer idle timeouts; corrupted multipart uploads; disk-full on local storage backends.","solutions":["Retry the parse job (asynq already allows MaxRetry 2) — transient stream errors often resolve","Check network stability between worker and storage backend (proxies, idle timeouts)","Verify object integrity (checksum/size) in storage to rule out a truncated upload","If the +1 byte LimitReader overflow is the issue, enforce the size limit earlier at upload time"],"exampleFix":"// before\ndata, err := io.ReadAll(io.LimitReader(file, secutils.GetMaxFileSizeMB()*1024*1024+1))\nif err != nil {\n    return \"\", nil, nil, fmt.Errorf(\"read source file: %w\", err)\n}\n// after\ndata, err := io.ReadAll(io.LimitReader(file, secutils.GetMaxFileSizeMB()*1024*1024+1))\nif err != nil {\n    logger.Warn(\"parse: read failed, may retry\", \"doc\", document.ID, \"err\", err)\n    return \"\", nil, nil, fmt.Errorf(\"read source file: %w\", err)\n}","handlingStrategy":"retry","validationCode":"if size > int64(secutils.GetMaxFileSizeMB())*1024*1024 { return ErrFileTooLarge }","typeGuard":null,"tryCatchPattern":"err := svc.Process(ctx, task)\nif err != nil && strings.HasPrefix(err.Error(), \"read source file:\") {\n    return err // transient IO error: let asynq retry (MaxRetry 2)\n}","preventionTips":["Raise proxy/load-balancer idle timeouts for large object streams","Verify upload checksums so truncated objects are rejected early","Enforce max file size at upload, not only at parse","Retry transient IO failures with backoff before marking documents failed"],"tags":["io-error","file-storage","parsing"],"backgroundTag":"source-file-read-failed","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}