Introduction

In the previous article, we built the Knowledge Document Ingestion Pipeline.

The platform can now:

• Upload company documents
• Validate files
• Extract text from PDFs, Word documents, Markdown, and text files
• Perform OCR on scanned documents
• Clean extracted text
• Normalize formatting
• Detect document language
• Extract metadata
• Store clean document text

This is a major milestone.

However, one important problem remains.

Large Language Models and vector databases are not designed to efficiently search an entire 250-page proposal, policy manual, or technical specification every time a user asks a question.

Sending an entire document to an LLM would:

• Consume too many tokens
• Increase latency
• Increase AI costs
• Reduce retrieval accuracy
• Introduce irrelevant context

Instead, documents must be divided into smaller, semantically meaningful units called chunks.

A well-designed chunking strategy is one of the most important factors in the quality of a Retrieval-Augmented Generation (RAG) system.

Poor chunks produce poor retrieval.

Excellent chunks produce excellent proposals.

In this article we will build the production-ready chunking engine that prepares every knowledge document for semantic search and AI-powered proposal generation.

Development Progress

Proposal Generation Architecture     ████████████████████ 100%
Organization Knowledge Base          ████████████████████ 100%
Knowledge Ingestion Pipeline         ████████████████████ 100%
Knowledge Chunking Engine            ████████████████████ 100%
Embedding Generation                 ░░░░░░░░░░░░░░░░░░░░   0%
Vector Database                      ░░░░░░░░░░░░░░░░░░░░   0%
Semantic Search                      ░░░░░░░░░░░░░░░░░░░░   0%
Retrieval-Augmented Generation       ░░░░░░░░░░░░░░░░░░░░   0%
Proposal Templates                   ░░░░░░░░░░░░░░░░░░░░   0%
Proposal Section Generator           ░░░░░░░░░░░░░░░░░░░░   0%
Proposal Review                      ░░░░░░░░░░░░░░░░░░░░   0%
Proposal Export                      ░░░░░░░░░░░░░░░░░░░░   0%

Why Chunking Is Necessary

Suppose a company uploads a 180-page Information Security Policy.

A tender asks:

“Describe your incident response process.”

Without chunking:

Entire 180-page document
↓
LLM
↓
Huge context
↓
High cost
↓
Lower accuracy

With chunking:

Incident Response Chunk
↓
Retrieved
↓
LLM
↓
Focused answer

The AI only receives relevant knowledge.

Chunking Goals

The chunking engine should:

• Preserve meaning
• Preserve context
• Support semantic search
• Support citations
• Support page references
• Minimize token usage
• Improve retrieval precision
• Remain deterministic
• Support versioning
• Scale to millions of chunks

What Is a Chunk?

A chunk is the smallest independently retrievable piece of knowledge.

Example:

Section 4.2
Incident Response
Our organization maintains a
24-hour incident response team...
End Section

The chunk should remain understandable without requiring the entire document.

Bad Chunk Example

"...therefore the implementation..."
"...all procedures..."
"...the following..."

This lacks context.

Good Chunk Example

Incident Response
Our organization operates a
24-hour incident response process
covering detection,
classification,
containment,
eradication,
recovery
and post-incident review.

The heading provides semantic meaning.

Chunking Architecture

Extracted Text
↓
Document Parser
↓
Section Detection
↓
Chunk Generator
↓
Chunk Validation
↓
Chunk Storage
↓
Embedding Generation

Chunking Pipeline

Load Extracted Text
↓
Identify Structure
↓
Split Sections
↓
Merge Small Sections
↓
Apply Overlap
↓
Generate Metadata
↓
Persist

Folder Structure

Create:

app/
knowledge/
chunking/
    parser/
    splitter/
    validator/
    metadata/
    services/

Chunk Model

Each chunk represents one logical knowledge fragment.

Knowledge Document
↓
Version
↓
Chunk 1
Chunk 2
Chunk 3
Chunk 4

Database Schema

Create:

knowledge_chunks

Recommended fields:

id
organization_id
document_id
document_version_id
chunk_number
heading
content
page_start
page_end
token_count
character_count
language
metadata
created_at

Entity Relationships

Knowledge Document
↓
Version
↓
Chunks
↓
Future Embeddings

SQLAlchemy Model

Create:

app/models/knowledge_chunk.py

class KnowledgeChunk(Base):
    __tablename__ = (
        "knowledge_chunks"
    )
    id = mapped_column(
        primary_key=True
    )
    organization_id = mapped_column(
        ForeignKey(
            "organizations.id"
        ),
        index=True
    )
    document_id = mapped_column(
        ForeignKey(
            "knowledge_documents.id"
        )
    )
    document_version_id = mapped_column(
        ForeignKey(
            "knowledge_document_versions.id"
        )
    )
    chunk_number = mapped_column()
    heading = mapped_column(
        String(300)
    )
    content = mapped_column(
        Text
    )
    token_count = mapped_column()
    character_count = mapped_column()
    page_start = mapped_column()
    page_end = mapped_column()

Why Store Heading Separately?

Suppose a tender asks:

“Describe your disaster recovery strategy.”

The heading:

Disaster Recovery

provides additional retrieval signal.

Embedding both:

• heading
• body

improves semantic search.

Chunk Metadata

Each chunk should store:

Heading
Section Level
Page Range
Language
Chunk Number
Document Version
Department
Category

This metadata supports advanced filtering during retrieval.

Section Detection

Documents usually contain:

Heading
Paragraph
Subheading
Paragraph
Bullet List

The parser identifies this hierarchy before chunking.

Heading Detection

Common heading indicators:

#
##
1.
1.1
1.2
SECTION
CHAPTER

These patterns vary by document type but provide useful structural cues.

Paragraph Detection

Paragraphs remain the primary semantic unit.

Avoid splitting sentences across chunks whenever possible.

Lists

Preserve:

•
-
1.
2.

Lists often contain important compliance information.

Tables

Convert tables into readable text.

Example:

Requirement
Response
↓
Requirement: ISO 27001
Response: Certified

Chunk Size

Too small:

80 tokens

Too large:

3000 tokens

Recommended target:

400–800 tokens

This balances retrieval quality and LLM context usage.

Overlapping Chunks

Chunks should overlap slightly.

Example:

Chunk A
Sentence 1
Sentence 2
Sentence 3
Sentence 4
Chunk B
Sentence 3
Sentence 4
Sentence 5
Sentence 6

Overlap preserves context across boundaries.

Overlap Size

Typical overlap:

50–100 tokens

The exact value should be configurable.

Heading-Aware Chunking

Never separate:

Heading
↓
Paragraph

Instead:

Heading
Paragraph
Paragraph

Each heading should remain attached to its content.

Semantic Boundaries

Chunk boundaries should prefer:

• Section endings
• Paragraph endings
• Bullet groups
• Table boundaries

Avoid splitting inside sentences.

Chunk Generator

Create:

chunk_generator.py

Generator Interface

class ChunkGenerator:
    def generate(
        self,
        text,
    ):
        ...

Chunk Context

@dataclass
class ChunkContext:
    heading: str
    page_start: int
    page_end: int
    language: str
    chunk_number: int

Chunk Object

@dataclass
class GeneratedChunk:
    heading: str
    content: str
    metadata: dict

Token Estimation

Every chunk estimates token count.

Why?

Because embedding cost depends on tokens.

Store:

Estimated Tokens
Actual Characters

Token Estimator

class TokenEstimator:
    def estimate(
        self,
        text,
    ):
        return len(text) // 4

A proper tokenizer will replace this heuristic later.

Chunk Validation

Reject chunks that are:

Empty
Too Short
Too Large
Duplicate
Unreadable

Duplicate Detection

Some PDFs repeat:

Header
Footer
Legal Notice

Validator removes duplicate chunks.

Language Validation

Every chunk should inherit the document language.

Mixed-language documents may later receive more advanced handling.

Chunk Repository

Create:

knowledge_chunk_repository.py

Responsibilities:

• Store chunks
• Query chunks
• Delete version chunks
• Count chunks

Chunk Service

Workflow:

Load Text
↓
Generate Chunks
↓
Validate
↓
Persist
↓
Queue Embeddings

Storage Workflow

Delete Existing Chunks
↓
Insert New Chunks
↓
Commit

Version-specific chunks simplify updates.

Background Job

Text Ready
↓
Chunk Job
↓
Embedding Job

Each processing stage remains independent.

Quality Metrics

Measure:

Average Tokens
Average Characters
Chunk Count
Validation Failures
Duplicates Removed

Chunk Statistics

Example:

Document
178 Pages
↓
612 Chunks
↓
Average
540 Tokens

Logging

{
  "event":"chunk_generation_completed",
  "document":22,
  "chunks":611,
  "duration_ms":1843
}

Monitoring

Dashboard:

Chunk Jobs
Average Size
Largest Chunk
Validation Failures
Average Duration

React Status

Users see:

Uploaded
↓
Extracted
↓
Chunking
↓
Embedding
↓
Ready

Security

Chunk data inherits all security from the parent document.

Never expose:

• Another organization’s chunks
• Internal metadata
• Processing artifacts

Future Retrieval

Chunks become searchable:

Tender Question
↓
Semantic Search
↓
Relevant Chunks
↓
LLM
↓
Proposal

Testing

Create:

tests/
knowledge/
test_chunk_generator.py
test_validator.py
test_repository.py

Example Test

def test_chunk_size():
    chunks = generate_chunks(
        sample_text
    )
    assert len(chunks) > 0

Overlap Test

Verify:

Chunk A
↓
Overlap
↓
Chunk B

Current Deliverables

✔ Chunk architecture
✔ SQLAlchemy model
✔ Chunk generator
✔ Heading-aware chunking
✔ Overlap strategy
✔ Metadata
✔ Validation
✔ Statistics
✔ Background processing
✔ Repository
✔ Service
✔ Monitoring
✔ Testing

The platform now structures company knowledge into AI-friendly semantic units.

Recommended Git Commit

git add .
git commit -m "Implement knowledge chunking engine"
git push origin main

What We Will Build Next

The platform now stores structured knowledge chunks.

However, chunks remain plain text.

To enable semantic search, every chunk must be transformed into a numerical vector representation that captures its meaning rather than just its keywords.

The next article in the series is:

“Building the Embedding Generation Pipeline: Creating Semantic Representations of Organizational Knowledge.”

In that article we will implement:

• Embedding model selection
• Embedding service abstraction
• Batch embedding generation
• Vector normalization
• Embedding versioning
• Retry and failure handling
• Cost optimization
• Background embedding workers
• Embedding quality monitoring
• Database schema updates
• API integration
• Testing and benchmarking

This pipeline will prepare every knowledge chunk for storage in a vector database and enable high-quality semantic retrieval for AI-powered proposal generation.

Conclusion

The Knowledge Chunking Engine transforms extracted document text into structured, semantically meaningful units that are optimized for retrieval rather than human reading. By respecting headings, paragraphs, lists, and logical document boundaries, the platform preserves context while reducing the amount of information that must be processed during proposal generation.

Configurable chunk sizes, controlled overlap, metadata enrichment, validation, and background processing ensure that the resulting knowledge is consistent, searchable, and scalable. Each chunk is versioned alongside its source document, maintaining traceability and supporting future citations within generated proposals.

With structured chunks now available, the foundation has been laid for semantic search. The next stage converts every chunk into a vector embedding, allowing the platform to retrieve organizational knowledge based on meaning instead of simple keyword matching.