PHASE 1 — DOCUMENT & KNOWLEDGE CONTEXT RECONSTRUCTION 1.1 Document Purpose The documents collectively represent an HR policy knowledge environment. Primary purposes: - Define employee leave rules. - Explain remote-work requirements. - Define attendance procedures. - Answer frequently asked HR questions. - Provide procedural instructions. 1.2 Knowledge Categories The knowledge can be divided into: Knowledge Category| Example Policy| Leave eligibility Procedure| How to request leave Eligibility| Who qualifies for a benefit Restriction| When remote work is not permitted Documentation| Medical certificate requirement Exception| Special approval conditions Definition| Meaning of a leave category FAQ| Common employee questions Versioned knowledge| 2026 policy vs older policy 1.3 Stable vs Frequently Changing Knowledge Stable - Definitions - General HR terminology - Basic procedural concepts Frequently Changing - Leave entitlements - Approval rules - Remote-work limits - Policy effective dates - Required forms - Exception rules 1.4 Critical Knowledge High-priority information includes: - Eligibility - Number of permitted leave days - Approval requirements - Required documentation - Effective dates - Exceptions - Role-specific permissions 1.5 Knowledge That Should Not Be Blindly Ingested Examples: - Decorative pages - Repeated footers - Navigation elements - Duplicate FAQ answers - Old superseded policies - Unrelated appendices These may still be retained in the source repository but should not automatically become active retrieval content. 1.6 Agent Knowledge Requirement The agent needs to answer: «What does the current applicable policy say about this employee's question?» Therefore retrieval must consider: - Topic - Policy version - Effective date - Employee role - Access permissions - Document status --- PHASE 2 — DOCUMENT FORENSICS & STRUCTURE ANALYSIS 2.1 Expected Document Structure Example: Employee Leave Policy 2026 │ ├── 1. Introduction │ ├── 2. Definitions │ ├── 2.1 Annual Leave │ ├── 2.2 Sick Leave │ └── 2.3 Emergency Leave │ ├── 3. Eligibility │ ├── 4. Leave Entitlements │ ├── 5. Leave Application Procedure │ ├── 6. Approval Rules │ ├── 7. Exceptions │ └── Appendix A 2.2 Elements To Detect The extraction system should identify: - Document title - Heading hierarchy - Paragraphs - Tables - Numbered procedures - Bulleted lists - Definitions - Warnings - Exceptions - Footnotes - References - Appendices 2.3 Page Furniture Detection Repeated elements such as: Company HR Department Employee Policy Manual Page 12 of 35 should be identified as repeated page furniture when they do not contain substantive knowledge. 2.4 Structural Uncertainty If a scanned PDF contains unclear headings: SOURCE CONTENT → OCR TEXT INFERRED STRUCTURE → Heading candidate STATUS → REQUIRES VALIDATION The system should not silently treat an uncertain OCR heading as authoritative structure. --- PHASE 3 — EXTRACTION & OCR QUALITY ANALYSIS 3.1 Extraction Strategy Use document-specific extraction. Native PDF First attempt: PDF → Native text extraction → Layout analysis → Structure detection Scanned PDF Use: Scanned PDF → OCR → Layout reconstruction → OCR confidence analysis → Validation DOCX Use: DOCX → Paragraph extraction → Heading extraction → Table extraction → List extraction → Metadata extraction 3.2 OCR Risks Potential problems: - "0" interpreted as "O" - "1" interpreted as "I" - Missing punctuation - Incorrect dates - Incorrect percentages - Broken tables - Incorrect reading order Example: Source: Leave entitlement: 18 days OCR: Leave entitlement: 8 days This is a critical transformation error. Therefore numeric policy information should receive additional validation. 3.3 Extraction Status Each content unit should have a status such as: extraction_status: extracted OCR_extracted partially_extracted extraction_failed requires_validation 3.4 Critical Validation Higher validation priority: - Numbers - Dates - Policy limits - Eligibility conditions - Exceptions - Table values - Legal/compliance language --- PHASE 4 — CONTENT CLEANING & NORMALIZATION 4.1 Cleaning Rules Remove or normalize: - Excessive whitespace - Broken page breaks - Repeated headers - Repeated footers - OCR spacing errors - Unnecessary decorative characters 4.2 Preserve Do NOT remove: - Numbered steps - Policy conditions - Exceptions - Definitions - Tables - Units - Dates - Percentages - Warnings - References 4.3 Example Raw: Employees must submit leave requests at least 3 working days before the start date. Normalized: Employees must submit leave requests at least 3 working days before the start date. Meaning remains unchanged. 4.4 Duplicate Detection If two documents contain identical paragraphs: Document A → Section 4 Document B → Section 7 the system should identify possible duplication. It should not automatically delete one source. Instead: duplicate_group_id = DUP-0042 and preserve both provenance records. --- PHASE 5 — SEMANTIC STRUCTURE RECONSTRUCTION The system should reconstruct: DOCUMENT ↓ SECTION ↓ SUBSECTION ↓ TOPIC ↓ KNOWLEDGE UNIT 5.1 Example Document: Employee Leave Policy 2026 Section: 4. Leave Entitlements Subsection: 4.2 Sick Leave Topic: Sick Leave Eligibility Knowledge Unit: Employees meeting the defined eligibility requirements may apply for sick leave according to the current policy. 5.2 Relationships Example: Sick Leave ├── Definition ├── Eligibility ├── Entitlement ├── Application Procedure ├── Documentation Requirement └── Exception 5.3 Relationship Types Use controlled relationship labels: PARENT_OF CHILD_OF DEFINES REQUIRES EXCEPTION_TO REFERENCES RELATED_TO SUPERSEDES These relationships should only be created when supported by the source or clearly documented structural inference. --- PHASE 6 — KNOWLEDGE UNIT & CHUNKING ARCHITECTURE A single fixed chunk size should not be applied to every document. 6.1 Recommended Strategy Use: Heading-aware + semantic + parent-child chunking Pipeline: Document → Heading Detection → Semantic Sections → Knowledge Units → Child Chunks → Parent Context 6.2 Chunk Types Parent Chunk Contains broader context. Example: Section: 4. Leave Entitlements Child Chunk Contains a focused knowledge unit. Example: 4.2 Sick Leave Eligibility and entitlement rules... 6.3 Target Chunk Size Initial design target: Target: 300-700 tokens Minimum: ~150 tokens Maximum: ~900 tokens These are starting parameters, not universal constants. They should be adjusted using retrieval evaluation. 6.4 Chunk Boundary Rules Prefer boundaries at: - Heading changes - Paragraph groups - Procedure boundaries - Table boundaries - Definition boundaries - Exception boundaries Avoid splitting: Condition + Exception when separating them would change interpretation. 6.5 Overlap Use limited semantic overlap rather than blindly applying a large character overlap. Example: Previous chunk: Eligibility + beginning of entitlement Next chunk: Entitlement + exception Only include overlap when it improves boundary continuity. --- PHASE 7 — TABLE, LIST & STRUCTURED-CONTENT HANDLING Tables require special treatment. 7.1 Example Table Leave Type| Eligibility| Maximum Days| Approval Annual| Eligible employees| Policy-defined| Manager Sick| Eligible employees| Policy-defined| Manager/HR Emergency| Defined conditions| Policy-defined| Manager Do not flatten this into an unstructured sequence such as: Annual Eligible employees Policy-defined Manager Sick... That would weaken relationships between columns. 7.2 Structured Representation Represent each row logically: leave_type = Annual eligibility = Eligible employees maximum_days = Policy-defined approval = Manager source = Leave Policy 2026 7.3 Table Storage Recommended: Original table + Normalized table representation + Source coordinates + Parent section 7.4 Procedures Numbered procedures should remain ordered. Example: 1. Submit request. 2. Attach required documentation. 3. Select leave dates. 4. Submit for approval. 5. Await approval. The ordering is part of the meaning. --- PHASE 8 — METADATA & PROVENANCE ARCHITECTURE Every knowledge unit should retain traceability. 8.1 Metadata Example { "document_id": "HR-LEAVE-2026", "document_title": "Employee Leave Policy 2026", "version": "2026.1", "source_type": "PDF", "page": 12, "section": "4.2 Sick Leave", "content_type": "policy", "topic": "sick leave", "effective_date": "2026-01-01", "status": "active", "extraction_status": "validated", "access_scope": "employee", "parent_chunk_id": "HR-LEAVE-P04", "source_location": "page_12", "confidence": "high" } 8.2 Important Distinction Metadata must distinguish: SOURCE CONTENT from: GENERATED METADATA For example: Source says: "Sick leave..." Generated metadata: topic = sick_leave The generated topic must not be presented as if it were source text. --- PHASE 9 — ENTITY & KNOWLEDGE RELATIONSHIP MAPPING Potential entities: Employee Manager HR Department Annual Leave Sick Leave Remote Work Attendance Policy Version Approval Required Document 9.1 Example Relationship Sick Leave ↓ requires ↓ Medical Documentation Another: Leave Policy 2026 ↓ supersedes ↓ Leave Policy 2025 Only create the "SUPERSEDES" relationship when the source metadata or document lifecycle explicitly establishes it. 9.2 Entity Provenance Every important extracted entity should be traceable to: document_id page section source text --- PHASE 10 — RETRIEVAL ARCHITECTURE Recommended pipeline: USER QUERY ↓ QUERY ANALYSIS ↓ ACCESS-CONTROL FILTER ↓ METADATA FILTER ↓ HYBRID RETRIEVAL ↓ RERANKING ↓ PARENT-CONTEXT EXPANSION ↓ DEDUPLICATION ↓ SOURCE VALIDATION ↓ CONTEXT ASSEMBLY ↓ AGENT 10.1 Vector Retrieval Useful for: "What are the rules for taking medical leave?" when the user does not use the exact wording from the document. 10.2 Keyword Retrieval Useful for: "18 days" "Form HR-07" "Policy 2026.1" Exact terms, identifiers, dates and codes may benefit from keyword search. 10.3 Hybrid Search Use: Semantic similarity + Keyword matching + Metadata filtering This is appropriate for the HR policy scenario. 10.4 Reranking Use reranking when the initial retrieval produces several semantically similar policy passages and exact applicability matters. --- PHASE 11 — CONTEXT ASSEMBLY & AGENT CONSUMPTION Retrieved information should not simply be concatenated. 11.1 Context Assembly Example: Primary evidence + Parent section + Relevant exception + Related table row + Source metadata 11.2 Ordering Recommended order: 1. Most directly relevant knowledge unit 2. Applicable exception 3. Supporting parent context 4. Related procedure 5. Source information 11.3 Deduplication If three chunks contain essentially the same rule: Chunk A Chunk B Chunk C the context assembler should avoid sending all three unless they provide distinct provenance or meaning. 11.4 Conflict Handling Suppose: Policy 2025: Annual leave = X Policy 2026: Annual leave = Y The system should identify: Different versions and use: effective_date + status + version rather than mixing the two. --- PHASE 12 — VERSIONING, FRESHNESS & KNOWLEDGE LIFECYCLE Lifecycle: INGEST ↓ IDENTIFY VERSION ↓ EXTRACT ↓ CHUNK ↓ INDEX ↓ VALIDATE ↓ PUBLISH For an update: NEW VERSION ↓ COMPARE ↓ RE-CHUNK AFFECTED CONTENT ↓ RE-INDEX ↓ MARK OLD VERSION SUPERSEDED ↓ VALIDATE ↓ PUBLISH 12.1 Document States Use: DRAFT ACTIVE SUPERSEDED EXPIRED DELETED QUARANTINED 12.2 Critical Rule A superseded policy should not remain silently available through normal retrieval. It can remain in the archival store for audit purposes while being excluded from active retrieval. --- PHASE 13 — ACCESS CONTROL & KNOWLEDGE SECURITY Authorization must occur before sensitive knowledge enters the agent context. 13.1 Example Employee → General policies Manager → General + manager policies HR Admin → General + manager + restricted HR policies 13.2 Retrieval Filter Example: user_role = employee AND access_scope IN ["employee", "public"] AND status = "active" 13.3 Important Security Principle Do NOT use: Vector similarity as authorization. Instead: Identity → Authorization → Metadata filtering → Retrieval 13.4 Deleted Documents If a document is deleted: Document status → DELETED Active retrieval → excluded Vector index → deletion/update job Cache → invalidated This prevents stale cached knowledge from remaining accessible. --- PHASE 14 — RAG QUALITY & EVALUATION No evaluation scores should be invented before testing. 14.1 Evaluation Dataset Create test cases such as: Exact Query How many annual leave days are allowed? Paraphrased Query How much paid vacation can an employee take? Multi-Hop Query Can a new employee take sick leave and what documentation is required? Ambiguous Query How much leave do I get? The system may need clarification because the leave type is unspecified. Missing Evidence Does the company provide 90 days of remote work? If the source does not contain this information, the agent should say that the available knowledge base does not establish it. Version Query What does the 2026 policy say? The system should explicitly retrieve the 2026 version. Table Query Who approves emergency leave? Retrieval should preserve the relationship between the leave type and approval column. 14.2 Metrics Measure: Retrieval Precision Retrieval Recall Context Relevance Context Completeness Citation Correctness Answer Faithfulness Source Coverage Stale Content Rate Duplicate Content Rate Retrieval Latency 14.3 Evaluation Principle Do not ask: "Did the vector database return something?" Ask: "Did retrieval return the evidence required to correctly answer the question?" --- PHASE 15 — KNOWLEDGE-BASE IMPLEMENTATION BLUEPRINT Component 1 — Ingestion Responsibility Receive documents and assign document identity. Input PDF, DOCX and other supported sources. Output Canonical source document. Failure Modes - Corrupted file - Unsupported format - Duplicate upload Validation File integrity and document identity checks. --- Component 2 — Extraction Responsibility Extract text, layout and structured content. Input Canonical document. Output Extracted content model. Failure Modes - OCR errors - Missing pages - Incorrect reading order Validation Extraction confidence and structural checks. --- Component 3 — Structure Parser Responsibility Reconstruct headings, sections and content relationships. Output Hierarchical document representation. --- Component 4 — Semantic Chunker Responsibility Create retrieval-oriented knowledge units. Output Parent and child chunks. Validation Boundary integrity and semantic completeness. --- Component 5 — Metadata Generator Responsibility Attach provenance and retrieval metadata. Output Metadata-enriched knowledge units. --- Component 6 — Embedding Responsibility Generate vector representations. Output Embedding vectors. --- Component 7 — Indexing Responsibility Store searchable knowledge. Output Vector and keyword indexes. --- Component 8 — Retrieval Responsibility Find candidate evidence. Output Ranked knowledge candidates. --- Component 9 — Context Assembly Responsibility Construct agent-ready evidence. Output Deduplicated contextual evidence. --- Component 10 — Access Control Responsibility Prevent unauthorized knowledge retrieval. Output Authorized evidence only. --- Component 11 — Version Manager Responsibility Maintain document lifecycle. Output Correct active knowledge state. --- Component 12 — Evaluation Responsibility Measure retrieval and answer quality. Output Evaluation metrics and failure reports. --- PHASE 16 — FINAL AGENT KNOWLEDGE BASE ARCHITECTURE PACK 1. Executive Summary The recommended architecture uses a semantic, hierarchy-aware, provenance-preserving RAG pipeline. Core design: Documents → Ingestion → Extraction/OCR → Cleaning → Structure Reconstruction → Semantic Chunking → Metadata → Hybrid Index → Permission Filtering → Retrieval → Reranking → Parent Context → Validation → Agent The architecture avoids blindly splitting documents into fixed-size chunks. It preserves: - Document hierarchy - Tables - Procedures - Definitions - Exceptions - Versions - Provenance - Access permissions --- 2. Knowledge Source Analysis Source| Type| Main Knowledge| Change Frequency| Priority Leave Policy 2026| PDF| Leave rules| High| Critical Remote Work Guidelines| PDF| Remote work| Medium/High| High Attendance Manual| DOCX| Attendance| Medium| High HR FAQ| PDF| Common questions| Medium| Medium --- 3. Document Structure Model Document ├── Metadata ├── Sections │ ├── Subsections │ │ ├── Topics │ │ │ └── Knowledge Units │ └── Tables ├── Procedures ├── Definitions ├── Exceptions ├── References └── Appendices --- 4. Extraction & OCR Strategy Native PDF → Native extraction Scanned PDF → OCR → OCR confidence → Layout reconstruction → Validation DOCX → Paragraphs → Headings → Tables → Lists Critical OCR fields such as dates, quantities and policy limits require additional validation. --- 5. Cleaning & Normalization Rules Remove: - Repeated headers - Repeated footers - Decorative elements - Page-number noise - Extraction artifacts Preserve: - Tables - Lists - Definitions - Exceptions - Numbers - Dates - Units - Procedures - References --- 6. Semantic Structure DOCUMENT → SECTION → SUBSECTION → TOPIC → KNOWLEDGE UNIT Relationships: PARENT_OF CHILD_OF DEFINES REQUIRES EXCEPTION_TO REFERENCES RELATED_TO SUPERSEDES --- 7. Chunking Strategy Primary strategy: Heading-aware semantic chunking with parent-child retrieval. Initial configuration: Target: 300-700 tokens Minimum: ~150 tokens Maximum: ~900 tokens Overlap: Only where semantic continuity requires it These parameters must be tuned through evaluation. --- 8. Structured Content Handling Content| Handling Paragraph| Semantic chunk Heading| Structure metadata Table| Structured representation + source table List| Preserve ordering Procedure| Preserve numbered sequence Definition| Dedicated knowledge unit Exception| Link to applicable rule Code| Preserve formatting Formula| Preserve mathematical structure FAQ| Question-answer knowledge unit --- 9. Metadata & Provenance Schema Metadata| Purpose document_id| Unique source identity document_title| Source identification version| Version control source_type| PDF/DOCX/etc. page| Page traceability section| Hierarchy subsection| Detailed hierarchy content_type| Policy/procedure/table/etc. topic| Retrieval classification effective_date| Freshness expiration_date| Lifecycle status| Active/superseded/etc. access_scope| Authorization parent_chunk_id| Parent context related_chunk_ids| Relationship mapping extraction_status| Extraction quality source_location| Traceability confidence| Validation state --- 10. Entity & Relationship Model Example: Entity: Sick Leave Attributes: eligibility entitlement documentation approval Relationships: Sick Leave → REQUIRES → Medical Documentation Sick Leave → APPROVED_BY → Manager Policy 2026 → SUPERSEDES → Policy 2025 Relationships must remain source-grounded. --- 11. Retrieval Architecture Query ↓ Query Classification ↓ Authorization ↓ Metadata Filtering ↓ Hybrid Search ↓ Candidate Retrieval ↓ Reranking ↓ Parent/Neighbor Expansion ↓ Deduplication ↓ Evidence Validation ↓ Context Assembly ↓ Agent --- 12. Context Assembly Strategy Context should contain: Primary Evidence + Required Parent Context + Relevant Exception + Relevant Table Data + Source Metadata Avoid: - Duplicate evidence - Superseded policies - Unauthorized information - Irrelevant chunks - Conflicting versions --- 13. Versioning & Knowledge Lifecycle INGEST ↓ IDENTIFY VERSION ↓ EXTRACT ↓ STRUCTURE ↓ CHUNK ↓ INDEX ↓ VALIDATE ↓ PUBLISH Update: NEW VERSION ↓ DIFF ↓ REPROCESS ↓ REINDEX ↓ SUPERSEDE OLD VERSION ↓ VALIDATE ↓ PUBLISH --- 14. Access Control & Security Authorization model: User Identity ↓ Role ↓ Document Permission ↓ Metadata Filter ↓ Retrieval ↓ Agent Context Security must apply before retrieval results enter the agent context. Vector similarity is not an authorization mechanism. --- 15. RAG Evaluation Framework Test Type| Example Exact| "How many leave days?" Paraphrase| "How much vacation can I take?" Multi-hop| Eligibility + documentation Ambiguous| "How much leave do I get?" Missing evidence| Unsupported company rule Version| "What does 2026 policy say?" Table| "Who approves emergency leave?" Cross-section| Rule + exception Metrics: - Retrieval Precision - Retrieval Recall - Context Relevance - Context Completeness - Citation Correctness - Answer Faithfulness - Source Coverage - Stale Content Rate - Duplicate Content Rate - Latency No scores should be claimed until the test set has actually been executed. --- 16. Knowledge Base Pipeline SOURCE DOCUMENTS | v INGESTION | v DOCUMENT FORENSICS | v EXTRACTION / OCR | v CLEANING / NORMALIZATION | v STRUCTURE RECONSTRUCTION | v SEMANTIC CHUNKING | v METADATA + PROVENANCE | v ENTITY RELATIONSHIPS | +---------+---------+ | | v v VECTOR INDEX KEYWORD INDEX | | +---------+---------+ | v HYBRID RETRIEVAL | v RERANKING | v PARENT CONTEXT EXPANSION | v DEDUPLICATION | v SOURCE VALIDATION | v CONTEXT ASSEMBLY | v AGENT | v CITED ANSWER --- 17. Knowledge Unit Specification Field| Purpose| Example/Rule| Required| Validation knowledge_unit_id| Unique identity| KU-0001| Yes| Unique document_id| Source mapping| HR-LEAVE-2026| Yes| Exists version| Version tracking| 2026.1| Yes| Valid version title| Knowledge identification| Sick Leave Eligibility| Yes| Source/derived section| Hierarchy| 4.2| Yes| Structure check content| Main knowledge| Extracted text| Yes| Source validation content_type| Content classification| Policy| Yes| Controlled vocabulary page| Provenance| 12| Yes| Page exists parent_chunk_id| Context| HR-PARENT-04| Recommended| Parent exists access_scope| Security| Employee| Yes| Permission check effective_date| Freshness| Policy date| Conditional| Date validation status| Lifecycle| Active| Yes| Lifecycle check extraction_status| Extraction quality| Validated| Yes| Extraction check confidence| Quality indicator| High| Recommended| Validation evidence embedding_id| Vector mapping| EMB-0001| Yes| Index check source_location| Exact traceability| Page/section| Yes| Source check --- 18. Retrieval Decision Matrix Query Type| Retrieval Strategy| Filters| Context Strategy| Validation Exact policy term| Keyword + Vector| Version/access| Exact chunk| Source match Natural-language question| Hybrid| Topic/access/status| Parent + child| Evidence check Policy number| Keyword| Version/access| Exact section| Identifier match Table question| Hybrid| Content type/table| Table + parent| Row/column check Procedure| Hybrid| Active version| Ordered procedure| Sequence validation Ambiguous question| Retrieval + clarification| User scope| Candidate evidence| Query clarification Version-specific| Metadata + hybrid| Version/effective date| Version-specific chunks| Version validation Multi-hop| Hybrid + reranking| Access/version| Multiple linked chunks| Cross-source validation Missing evidence| Retrieval| Access/status| Best candidates| Evidence sufficiency check --- 19. Implementation Sequence Step| Component| Dependency| Deliverable| Validation| Definition of Done 1| Source Registry| None| Document catalog| Identity check| All sources registered 2| Ingestion| Registry| Canonical files| Integrity check| Files accessible 3| Extraction| Ingestion| Extracted content| Extraction QA| Required content extracted 4| OCR| Extraction| OCR content| Confidence QA| Critical OCR validated 5| Structure Parser| Extraction| Document hierarchy| Structural QA| Sections reconstructed 6| Cleaner| Structure| Normalized content| Content comparison| Meaning preserved 7| Chunker| Structure| Knowledge units| Boundary QA| Chunks semantically complete 8| Metadata| Chunker| Metadata records| Schema validation| Required metadata present 9| Embeddings| Chunks| Vectors| Index check| Vectors generated 10| Search Index| Metadata + vectors| Search indexes| Query tests| Retrieval functional 11| Access Control| Metadata| Security filters| Authorization tests| Unauthorized data blocked 12| Reranking| Retrieval| Ranked candidates| Retrieval tests| Ranking improves evidence selection 13| Context Assembly| Retrieval| Agent context| Context tests| Relevant evidence assembled 14| Versioning| Source Registry| Lifecycle system| Version tests| Old content controlled 15| Evaluation| Complete pipeline| Evaluation dataset| Metric analysis| Quality measurable 16| Monitoring| Production pipeline| Operational dashboard| Alert tests| Failures observable --- 20. Final Agent Knowledge Base Blueprint Recommended Architecture ┌─────────────────────┐ │ SOURCE DOCUMENTS │ └──────────┬──────────┘ ↓ ┌─────────────────────┐ │ INGESTION │ └──────────┬──────────┘ ↓ ┌─────────────────────┐ │ DOCUMENT FORENSICS │ └──────────┬──────────┘ ↓ ┌─────────────────────┐ │ EXTRACTION / OCR │ └──────────┬──────────┘ ↓ ┌─────────────────────┐ │ CLEAN + NORMALIZE │ └──────────┬──────────┘ ↓ ┌─────────────────────┐ │ STRUCTURE MODEL │ └──────────┬──────────┘ ↓ ┌─────────────────────┐ │ SEMANTIC CHUNKING │ └──────────┬──────────┘ ↓ ┌──────────────┴──────────────┐ ↓ ↓ METADATA LAYER RELATIONSHIP LAYER │ │ └──────────────┬──────────────┘ ↓ ┌────────────────────┐ │ INDEXING LAYER │ │ Vector + Keyword │ └─────────┬──────────┘ ↓ ┌────────────────────┐ │ ACCESS CONTROL │ └─────────┬──────────┘ ↓ ┌────────────────────┐ │ HYBRID RETRIEVAL │ └─────────┬──────────┘ ↓ ┌────────────────────┐ │ RERANKING │ └─────────┬──────────┘ ↓ ┌────────────────────┐ │ CONTEXT ASSEMBLY │ └─────────┬──────────┘ ↓ ┌────────────────────┐ │ SOURCE CHECK │ └─────────┬──────────┘ ↓ ┌────────────────────┐ │ AI AGENT │ └─────────┬──────────┘ ↓ ┌────────────────────┐ │ CITED ANSWER │ └────────────────────┘ Final Design Principles 1. Source-first architecture Every important answer must be traceable to source material. 2. Semantic chunking Chunks represent meaningful knowledge units rather than arbitrary character ranges. 3. Hierarchy preservation Parent sections remain available when child chunks require additional context. 4. Structured-content preservation Tables, procedures, lists and definitions are handled according to their semantics. 5. Version awareness Current and historical policies are explicitly separated. 6. Permission-aware retrieval Authorization occurs before protected information reaches the agent. 7. Hybrid retrieval Vector search handles semantic similarity while keyword retrieval supports exact terms, identifiers and numbers. 8. Evidence-aware context Only relevant, authorized and sufficiently supported evidence should enter the agent context. 9. Traceable transformations Every major transformation can be traced back to the source. 10. Evaluation-driven optimization Chunk size, overlap, retrieval configuration and reranking should be adjusted using actual evaluation results rather than assumptions. End-to-End Knowledge Lifecycle DOCUMENT → INGEST → FORENSICS → EXTRACT → OCR IF REQUIRED → CLEAN → RECONSTRUCT STRUCTURE → SEMANTICALLY CHUNK → ATTACH METADATA → MAP RELATIONSHIPS → EMBED → INDEX → APPLY ACCESS CONTROL → RETRIEVE → RERANK → ASSEMBLE CONTEXT → VALIDATE SOURCES → ANSWER → EVALUATE → MONITOR → UPDATE → REINDEX Sample Test Result: The resulting system is not simply a PDF-to-text converter. It is a version-aware, permission-aware, provenance-preserving, semantic RAG knowledge architecture designed around how an AI agent actually retrieves and consumes evidence.