Tencent/WeKnora · error

read source file: %w

Error message

read source file: %w

What it means

After opening the source file, parse reads it with io.ReadAll under a LimitReader capped at the max file size (+1 byte to detect overflow); an I/O error during the read is wrapped as 'read source file'. This happens after a successful open, so the problem is mid-stream: connection reset, disk error, or truncated object.

Source

Thrown at internal/application/service/temporary_document.go:358

		})
	}
	chunksJSON, _ := json.Marshal(chunks)
	imagesJSON, _ := json.Marshal(images)
	metadataJSON, _ := json.Marshal(metadata)
	return s.repo.MarkReady(ctx, payload.TenantID, payload.DocumentID, content,
		types.JSON(chunksJSON), types.JSON(imagesJSON), types.JSON(metadataJSON),
		chunker.ApproxTokenCount(content, lang), len(chunks), time.Now())
}

func (s *temporaryDocumentService) parse(ctx context.Context, document *types.TemporaryDocument) (string, []types.TemporaryDocumentImage, map[string]string, error) {
	file, err := s.fileService.GetFile(ctx, document.ResourceRef)
	if err != nil {
		return "", nil, nil, fmt.Errorf("open source file: %w", err)
	}
	defer file.Close()
	data, err := io.ReadAll(io.LimitReader(file, secutils.GetMaxFileSizeMB()*1024*1024+1))
	if err != nil {
		return "", nil, nil, fmt.Errorf("read source file: %w", err)
	}
	ext := document.FileType
	var options types.TemporaryDocumentCreateOptions
	_ = json.Unmarshal(document.ProcessingOptions, &options)
	if options.ParserEngine == "" || options.ParserEngine == "auto" {
		if tenant, ok := ctx.Value(types.TenantInfoContextKey).(*types.Tenant); ok && tenant != nil {
			options.ParserEngine = tenant.ParserEngineConfig.ResolveChatParserEngine(ext)
		}
	}
	if _, ok := temporaryTextExtensions[ext]; ok && (options.ParserEngine == "" || options.ParserEngine == "auto") {
		return string(data), nil, map[string]string{"parser": "plain_text"}, nil
	}
	if docparser.IsAudioFormat(ext) {
		if options.ASRModelID == "" {
			return "", nil, nil, fmt.Errorf("audio transcription model is not configured")
		}
		asrModel, err := s.modelService.GetASRModel(ctx, options.ASRModelID)
		if err != nil {

View on GitHub (pinned to 988cbb0330)

Solutions

1. Retry the parse job (asynq already allows MaxRetry 2) — transient stream errors often resolve
2. Check network stability between worker and storage backend (proxies, idle timeouts)
3. Verify object integrity (checksum/size) in storage to rule out a truncated upload
4. If the +1 byte LimitReader overflow is the issue, enforce the size limit earlier at upload time

Example fix

// before
data, err := io.ReadAll(io.LimitReader(file, secutils.GetMaxFileSizeMB()*1024*1024+1))
if err != nil {
    return "", nil, nil, fmt.Errorf("read source file: %w", err)
}
// after
data, err := io.ReadAll(io.LimitReader(file, secutils.GetMaxFileSizeMB()*1024*1024+1))
if err != nil {
    logger.Warn("parse: read failed, may retry", "doc", document.ID, "err", err)
    return "", nil, nil, fmt.Errorf("read source file: %w", err)
}

if size > int64(secutils.GetMaxFileSizeMB())*1024*1024 { return ErrFileTooLarge }

err := svc.Process(ctx, task)
if err != nil && strings.HasPrefix(err.Error(), "read source file:") {
    return err // transient IO error: let asynq retry (MaxRetry 2)
}

When it happens

Trigger: Storage stream breaking mid-read (network drop to S3/GCS); underlying object truncated or corrupted; local disk read errors; timeout killing the connection partway through.

Common situations: Large files over flaky network paths; proxy/load-balancer idle timeouts; corrupted multipart uploads; disk-full on local storage backends.

Related errors

• open source file: %w
• decode suggestion JSON: %w
• parse document: %w
• attachment %s failed to parse: %s
• readability parse: %w

AI-assisted analysis of Tencent/WeKnora@988cbb0330 (2026-09-02). Data as JSON: /api/errors/616023e7e677c7c7. Report an issue: GitHub.