Introduction

In the previous article, we built the Organization Knowledge Base, providing a secure repository for company information that will later power AI-generated tender proposals.

Users can now:

• Upload company documents
• Organize knowledge into categories
• Manage document versions
• Review and approve content
• Secure knowledge with role-based permissions
• Maintain a governed source of organizational information

However, uploaded documents are still just files.

A PDF, Word document, or PowerPoint presentation cannot be directly used by a Large Language Model.

Before AI can retrieve useful knowledge, documents must first be transformed into structured, machine-readable content.

This process is known as document ingestion.

Document ingestion is responsible for:

• Reading uploaded files
• Extracting text
• Running OCR when necessary
• Cleaning formatting artifacts
• Detecting document language
• Extracting metadata
• Normalizing whitespace
• Identifying document structure
• Preparing content for chunking and embeddings

Without a reliable ingestion pipeline, every later AI capability—including Retrieval-Augmented Generation (RAG), semantic search, proposal generation, and compliance checking—will suffer from poor-quality input.

Garbage in produces garbage out.

In this article, we will build a production-ready ingestion pipeline that transforms uploaded company documents into clean, structured text suitable for downstream AI processing.

Development Progress

Proposal Generation Architecture     ████████████████████ 100%
Organization Knowledge Base          ████████████████████ 100%
Knowledge Document Ingestion         ████████████████████ 100%
Knowledge Chunking                   ░░░░░░░░░░░░░░░░░░░░   0%
Embeddings                           ░░░░░░░░░░░░░░░░░░░░   0%
Semantic Search                      ░░░░░░░░░░░░░░░░░░░░   0%
Retrieval-Augmented Generation       ░░░░░░░░░░░░░░░░░░░░   0%
Proposal Templates                   ░░░░░░░░░░░░░░░░░░░░   0%
Proposal Section Generation          ░░░░░░░░░░░░░░░░░░░░   0%
Compliance Matrix                    ░░░░░░░░░░░░░░░░░░░░   0%
Proposal Review                      ░░░░░░░░░░░░░░░░░░░░   0%
Proposal Export                      ░░░░░░░░░░░░░░░░░░░░   0%

Why Document Ingestion Is Important

Business documents are created for people—not machines.

A PDF may contain:

• Headers
• Footers
• Page numbers
• Logos
• Tables
• Images
• Watermarks
• Multi-column layouts
• Scanned pages
• Hidden formatting

Simply extracting raw text often produces poor results.

Example:

Page 7
CONFIDENTIAL
Company XYZ
ISO 27001 Policy
...
Page 8
CONFIDENTIAL
Company XYZ

Repeated page headers reduce retrieval quality.

The ingestion pipeline removes these artifacts before AI processing.

Overall Architecture

Uploaded Document
↓
Validation
↓
Malware Scan
↓
Text Extraction
↓
OCR (if required)
↓
Cleaning
↓
Normalization
↓
Metadata Extraction
↓
Language Detection
↓
Structured Text
↓
Knowledge Store

Every stage is independent and observable.

Processing Philosophy

Each stage performs exactly one responsibility.

Upload
↓
Extract
↓
Clean
↓
Normalize
↓
Analyze
↓
Store

This modular design makes the system easier to maintain and test.

Background Processing

Document ingestion should never occur inside the upload request.

Instead:

Upload
↓
Queue Job
↓
Return Response
↓
Worker Processes File

This keeps uploads responsive even for very large documents.

Processing Workflow

Document Version
↓
Malware Scan
↓
Determine File Type
↓
Extract Text
↓
Normalize
↓
Extract Metadata
↓
Store Clean Text
↓
Ready for Chunking

Supported File Types

The first production release will support:

PDF
DOCX
TXT
Markdown

Future versions may add:

PPTX
XLSX
HTML
ODT
RTF

Folder Structure

Create:

app/
knowledge/
    ingestion/
        extractors/
        cleaners/
        analyzers/
        metadata/
        storage/
        workers/
        pipelines/

Each component has a single responsibility.

Processing Pipeline

Create:

knowledge_ingestion_pipeline.py

Pipeline:

Load File
↓
Select Extractor
↓
Extract Text
↓
Clean Text
↓
Normalize
↓
Detect Language
↓
Extract Metadata
↓
Persist

Extractor Interface

from abc import ABC
from abc import abstractmethod
class DocumentExtractor(
    ABC,
):
    @abstractmethod
    def supports(
        self,
        mime_type: str,
    ) -> bool:
        ...
    @abstractmethod
    def extract(
        self,
        file_path: str,
    ) -> str:
        ...

All extractors implement the same interface.

Extractor Registry

class ExtractorRegistry:
    def __init__(self):
        self.extractors = []
    def register(
        self,
        extractor,
    ):
        self.extractors.append(
            extractor
        )
    def get(
        self,
        mime_type,
    ):
        for extractor in self.extractors:
            if extractor.supports(
                mime_type
            ):
                return extractor
        raise UnsupportedDocument()

PDF Extraction

Create:

pdf_extractor.py

Possible implementation:

class PDFExtractor(
    DocumentExtractor
):
    def supports(
        self,
        mime_type,
    ):
        return (
            mime_type
            == "application/pdf"
        )
    def extract(
        self,
        file_path,
    ):
        return extract_pdf_text(
            file_path
        )

The underlying extraction library can be swapped without changing the pipeline.

DOCX Extraction

class WordExtractor(
    DocumentExtractor
):
    def supports(
        self,
        mime_type,
    ):
        return (
            mime_type
            ==
            "application/vnd.openxmlformats-officedocument.wordprocessingml.document"
        )
    def extract(
        self,
        file_path,
    ):
        return extract_docx(
            file_path
        )

TXT Extraction

Simple implementation:

class TextExtractor(
    DocumentExtractor
):
    def supports(
        self,
        mime_type,
    ):
        return mime_type.startswith(
            "text/"
        )
    def extract(
        self,
        file_path,
    ):
        with open(
            file_path,
            encoding="utf-8",
        ) as file:
            return file.read()

Markdown Extraction

Markdown remains text.

The pipeline preserves headings while removing formatting artifacts only where appropriate.

Detecting Scanned PDFs

Some PDFs contain images instead of text.

Workflow:

Extract Text
↓
Text Empty?
↓
Yes
↓
Run OCR

OCR Pipeline

PDF
↓
Convert Pages
↓
OCR
↓
Merge Text
↓
Continue Processing

OCR should only run when necessary because it is significantly more expensive.

OCR Quality Threshold

Rather than checking only for empty text, estimate extraction quality.

Example:

Characters
Words
Alphabetic Ratio
Printable Characters

If quality is too low:

Fallback to OCR

Processing Status Updates

The version status changes throughout processing.

UPLOADED
↓
PROCESSING
↓
TEXT_EXTRACTED
↓
NORMALIZED
↓
READY_FOR_REVIEW

Failures update:

FAILED

Pipeline Context

Create:

@dataclass
class IngestionContext:
    organization_id: int
    document_version_id: int
    storage_key: str
    mime_type: str
    extracted_text: str | None
    metadata: dict

Each pipeline stage updates this shared context.

Cleaning Text

Raw extracted text contains noise.

Example:

Header
Company Name
Page 4
Confidential

Repeated on every page.

These elements should be removed.

Cleaning Pipeline

Raw Text
↓
Remove Headers
↓
Remove Footers
↓
Normalize Spaces
↓
Normalize Line Breaks
↓
Remove Empty Pages

Header Detection

Repeated lines appearing on many pages are likely headers.

Example:

Company XYZ
Quality Manual
Confidential

Frequency analysis can identify such patterns.

Footer Removal

Typical footers include:

Page 3 of 84
Confidential
Internal Use Only

These should not become searchable knowledge.

Whitespace Normalization

Normalize:

Tabs
Multiple Spaces
Blank Lines
Mixed Line Endings

Consistent formatting improves chunk generation.

Unicode Normalization

Normalize Unicode characters.

Examples:

Smart Quotes
Long Dashes
Non-breaking Spaces

Use NFC normalization where appropriate.

Hyphenation Repair

PDF extraction often produces:

organi-
zation

The cleaner repairs these words.

Bullet Detection

Preserve bullet structure.

Instead of:

•
•
•

Convert to consistent bullet formatting.

Table Handling

Tables are difficult.

Instead of preserving layout:

Name
Value

Represent logically:

Name: Value

Complex tables may require specialized processing in future releases.

Image Captions

Image captions often contain useful information.

Preserve:

Figure 2
System Architecture

Ignore decorative images.

Metadata Extraction

Extract metadata separately.

Examples:

Title
Author
Creation Date
Revision
Keywords

Do not rely exclusively on embedded document metadata because it is often missing or outdated.

Language Detection

Knowledge may exist in multiple languages.

Pipeline:

Extracted Text
↓
Language Detector
↓
ISO Language Code

Example:

en
nl
fr
de

Confidence Scores

Language detection should return:

Language
Confidence

Very short documents may require manual confirmation.

Reading Level

Later AI prompts may benefit from:

Reading Complexity
Sentence Length
Paragraph Length

Store these metrics as optional metadata.

Document Statistics

Useful statistics:

Characters
Words
Paragraphs
Pages
Tables
Images

These metrics also help estimate AI costs.

Processing Service

Create:

knowledge_ingestion_service.py

Service Flow

class KnowledgeIngestionService:
    def process(
        self,
        context,
    ):
        extractor = (
            self.registry.get(
                context.mime_type
            )
        )
        text = extractor.extract(
            context.storage_key
        )
        cleaned = self.cleaner.clean(
            text
        )
        metadata = (
            self.metadata.extract(
                cleaned
            )
        )
        context.extracted_text = (
            cleaned
        )
        context.metadata = metadata
        return context

Each service remains independently testable.

Text Storage

Store extracted text separately from the original file.

Benefits:

• Faster retrieval
• Simpler chunking
• Reduced repeated extraction
• Easier debugging

Storage Layout

Original File
↓
Object Storage
Extracted Text
↓
Separate Object
Metadata
↓
Database

Text Versioning

Every document version owns its extracted text.

Version 1
↓
Extracted Text v1
Version 2
↓
Extracted Text v2

Never overwrite older extracted text.

Failure Handling

Failures should be categorized.

Examples:

Unsupported File
Corrupt PDF
OCR Failure
Timeout
Storage Error
Unexpected Exception

Each category supports different retry behavior.

Retry Strategy

Retry:

Temporary Storage Failure
Network Error
OCR Timeout

Do not retry:

Unsupported Format
Invalid File
Permission Error

Logging

Each pipeline stage emits structured logs.

Example:

{
  "event": "knowledge_text_extracted",
  "document_version": 17,
  "pages": 94,
  "language": "en",
  "duration_ms": 2870
}

Metrics

Track:

Documents Processed
OCR Usage
Extraction Time
Failure Rate
Average Pages
Average Words

Dashboard

Operations dashboard:

Queued
Running
Completed
Failed
OCR Rate
Average Duration

React Status

Users see:

Uploading
↓
Processing
↓
Extracting Text
↓
Cleaning
↓
Ready for Review

Real-time progress improves user experience.

Security

Never expose extracted text from another organization.

Tenant isolation applies to:

• Original file
• Extracted text
• Metadata
• Processing jobs

Future Integration

The output of this article becomes the input for:

Chunking
↓
Embeddings
↓
Semantic Search
↓
RAG
↓
Proposal Generation

Testing

Create:

tests/
knowledge/
test_pdf_extractor.py
test_docx_extractor.py
test_cleaner.py
test_language.py
test_pipeline.py

Example Test

def test_whitespace_cleaning():
    raw = "Hello     World"
    cleaned = clean_text(raw)
    assert cleaned == "Hello World"

OCR Test

Verify:

Scanned PDF
↓
OCR Triggered
↓
Text Produced

Pipeline Test

Document
↓
Extract
↓
Normalize
↓
Metadata
↓
Store
↓
Success

Every stage should be validated independently.

Current Deliverables

✔ Modular ingestion pipeline
✔ Extractor architecture
✔ PDF extraction
✔ DOCX extraction
✔ TXT extraction
✔ OCR fallback
✔ Cleaning pipeline
✔ Metadata extraction
✔ Language detection
✔ Statistics
✔ Background processing
✔ Processing context
✔ Error handling
✔ Logging
✔ Metrics
✔ Progress reporting
✔ Testing

The platform can now transform uploaded company documents into clean, machine-readable knowledge.

Recommended Git Commit

git add .
git commit -m "Implement knowledge document ingestion pipeline"
git push origin main

What We Will Build Next

The ingestion pipeline now produces clean, normalized document text.

However, Large Language Models cannot efficiently search an entire 300-page policy manual or a 150-page case study every time a proposal is generated.

The next step is to divide documents into meaningful pieces that preserve context while remaining efficient for semantic search.

The next article in the series is:

“Building the Knowledge Chunking Engine: Structuring Documents for Semantic Search and AI Retrieval.”

In that article, we will implement:

• Intelligent document chunking
• Heading-aware segmentation
• Paragraph grouping
• Overlapping chunks
• Chunk metadata
• Chunk identifiers
• Page references
• Section hierarchy
• Token estimation
• Chunk validation
• Storage schema
• Background chunking jobs
• Quality metrics
• Testing

This chunking engine will become the foundation for embedding generation and Retrieval-Augmented Generation (RAG).

Conclusion

A reliable document ingestion pipeline is essential for any AI system that depends on organizational knowledge. Rather than treating uploaded files as opaque attachments, the pipeline transforms PDFs, Word documents, text files, and Markdown into structured, normalized content that can be processed consistently by downstream AI services.

By separating extraction, cleaning, normalization, metadata analysis, and storage into independent stages, the system remains modular, testable, and resilient. OCR fallback ensures scanned documents remain usable, while language detection, document statistics, and structured metadata provide valuable context for future retrieval and proposal generation.

With clean text now available, the platform is ready to move beyond document storage and begin understanding document structure. The next step is to divide extracted content into semantically meaningful chunks that can be embedded, searched, and retrieved with high precision during AI-powered proposal generation.