Own Your Knowledge Base: A Self-Hosted RAG Architecture

Many enterprise AI products follow the same pattern. You upload your documents to a cloud service, pay per page or per query, and receive answers from a system you cannot inspect.

For regulated industries such as law, insurance, and healthcare, this model creates three specific problems.

Data control. Documents leave your infrastructure. For organisations subject to data residency or confidentiality requirements, this is a compliance exposure.
Scaling cost. Cost scales with every document. Growth becomes an expense multiplier rather than an efficiency gain.
Auditability. When the system produces an incorrect answer on critical data, you have no visibility into why, and no way to trace or verify the logic.

We built a reference architecture to show a different approach: a document intelligence pipeline where all critical data processing runs locally, using open-source components. No data leaves your infrastructure. Every step is visible.

The Architecture

Each component was selected for a specific function: parsing, embedding, retrieval, and reasoning. All data processing runs locally on your hardware.

1. Document Parsing: Docling (IBM)

Standard OCR treats a document as a flat collection of text. It reads words, but it does not recognise structure.

The problem. If you process a complex regulatory PDF through a standard text extraction pipeline, it merges headers, footnotes, and tables. The original document hierarchy is lost.
The approach. We used Docling, an open-source IBM project that performs layout analysis. It identifies that bold text is a section header, and that the paragraph below it belongs to that concept.
The result. Structured, hierarchy-aware document chunks. The system reads a clause’s meaning from its position in the document, not only from the words on the page.

2. Local Embedding: EmbeddingGemma (Google)

Most RAG systems depend on external embedding APIs. Every document you embed is sent to a third-party server.

The model. We used EmbeddingGemma, Google’s open-source embedding model.
The implementation. We chose to access the model through the ONNX runtime, which allows it to run locally on CPU.
Why it matters. Your proprietary data stays on your own server. The mathematical fingerprint of your data never leaves your infrastructure.
The effect. High-quality semantic understanding without the latency or privacy exposure of an external API dependency.

3. Hybrid Search: PostgreSQL with pgvector

Vector search finds conceptually similar text. But in law or insurance, you often need an exact clause number or a specific regulatory term.

The stack. PostgreSQL with the pgvector extension.
The method. We implemented Reciprocal Rank Fusion (RRF). This technique merges multiple ranked lists into a single, more accurate result set. Each search produces two rankings at once: semantic ranking, using cosine similarity on vector embeddings; and keyword ranking, using PostgreSQL full-text search relevance.
The result. Exact phrase matches are prioritised when they exist, while the system still benefits from semantic understanding. This grounds every answer in real document excerpts.

4. The Interface: Pluggable AI Agent

The three components above form the pipeline. The AI agent is the reasoning layer that turns retrieved document chunks into answers.

The design principle. The agent is the only component that can be swapped without affecting your data.
Why this holds. In a RAG architecture, your memory, meaning the indexed documents and vector embeddings, is separated from your reasoning engine, meaning the model. Because the parsing and embedding logic is standardised, you can swap a cloud-based model for a local one without re-processing a single page.
The benefit. Your processed data stays under your control. The model sits on top of it as a replaceable component. If a model is deprecated or its pricing changes, you swap the agent. Your processed data is unaffected.

The Commercial Case: Rent or Own

Why invest the engineering effort instead of subscribing to a managed platform? Not always. But in some settings the trade-off favours owning the pipeline.

1. Fixed Cost Against Variable Expense

SaaS vendors charge per page. This is a variable cost that rises with your document volume.

With a locally hosted pipeline, the cost moves from operating expense to a fixed build and infrastructure cost. After the initial build, processing one million additional pages has near-zero marginal software cost. You control the throughput.

2. Auditable by Design

Every model can produce inaccurate results. The difference is how much visibility you have when it happens.

With a managed service. You typically see the answer but not the full retrieval logic, such as which documents were ranked or why certain passages were selected.
With a self-hosted pipeline. You control the retrieval layer. You can inspect retrieved chunks, filter weak matches before they reach the model, and log every query for compliance review. You control the quality of what goes in.

3. Vendor Independence

APIs change and models are deprecated.

This architecture separates the data layer, meaning your indexed documents, from the reasoning layer, meaning the model. Your processed, indexed data stays under your control regardless of which model you use.

The Trade-offs

Engineering expertise. This is not a self-service platform. It requires specialists who understand document processing and system architecture.
Operational responsibility. You own the system’s performance, reliability, and security.
Hardware requirements. Full local inference, including the model, requires GPU infrastructure. Our reference builds often use a hybrid approach: local document processing with a cloud model for the reasoning layer.

The Decision

Managed AI services are effective for general-purpose tasks. For many organisations they are the right choice.

But if your organisation operates in a regulated industry, where data residency and long-term cost predictability are requirements, the architecture of your AI pipeline is a decision worth making deliberately. You can rent that capability and accept the constraints on control, cost, and audit. Or you can host it yourself and keep those under your own control, at the cost of the engineering and operational responsibility that comes with it.

The question is which of those trade-offs fits your compliance obligations, your document volume, and the visibility your work requires.