Project Axiom: Unleashing Context with a Full-Stack RAG System

The quest for truly intelligent applications often boils down to one critical factor: context. Large Language Models (LLMs) are powerful, but their knowledge is finite and static. To bridge this gap and provide our applications with up-to-date, domain-specific, and highly relevant information, we embarked on Project Axiom.

Our goal was ambitious: build a complete Retrieval Augmented Generation (RAG) knowledge system from the ground up. This meant tackling everything from robust document ingestion and intelligent chunking to sophisticated hybrid search and seamless integration into our existing workflows. After an intense development session, I'm thrilled to share that the core infrastructure for Project Axiom is not just complete, but type-checking cleanly and ready for its maiden voyage.

Building the Foundation: Data and Storage

Any knowledge system begins with how it stores and manages information. For Axiom, we leveraged the power of Prisma and PostgreSQL to create a resilient and performant data layer:

• Prisma Models: We introduced three new models in prisma/schema.prisma:
  • ProjectDocument: To manage documents, including critical metadata like authority and category which are vital for nuanced search.
  • DocumentChunk: The atomic unit of our RAG system, each chunk storing its vector embedding for semantic search and a tsvector for full-text search.
  • AxiomApiToken: For secure external access, storing SHA-256 hashed tokens.
• PostgreSQL Powerhouse: Our database isn't just a data store; it's an intelligent search engine. We configured prisma/rls.sql to include:
  • Row-Level Security (RLS) policies for data isolation.
  • A vector column for storing embeddings.
  • An HNSW index for lightning-fast nearest-neighbor vector search.
  • A GIN FTS index for efficient full-text search.
  • A tsvector trigger to automatically update full-text search vectors.
• Expanded Storage: Our src/server/services/storage.ts adapter was extended to handle various document MIME types, with a generous 50MB maximum file size.

The Brains: Ingestion and Search

With the data foundation laid, the real intelligence of Axiom comes from how it processes and retrieves information.

Smart Document Processing (src/server/services/rag/document-processor.ts)

Not all documents are created equal, and neither should their chunking strategies be. Axiom employs intelligent, content-aware chunking:

• Markdown: Chunks are intelligently split by headings, preserving semantic structure.
• Plain Text: Paragraph-based chunking for readability.
• Code: Splits by declarations, ensuring code snippets remain coherent.
• PDF: Leverages pdf-parse v2 to extract text page-by-page, respecting the visual layout.

Sophisticated Hybrid Search (src/server/services/rag/document-search.ts)

Relying solely on vector similarity can sometimes miss explicit keywords, and vice-versa. Axiom combines the best of both worlds:

• Hybrid Approach: A weighted blend of 70% vector similarity search and 30% full-text search (tsvector). This ensures both semantic relevance and keyword accuracy.
• Authority Boosting: Critical for prioritizing information. Chunks from "mandatory" documents receive a significant +0.3 score boost, while "guideline" documents get +0.15. This ensures our LLMs always get the most authoritative answers first.

Contextual Content Loading (src/server/services/rag/load-axiom-content.ts)

Retrieving chunks is one thing; presenting them effectively to an LLM is another. This service ensures:

• Mandatory chunks are always loaded first.
• Relevant guideline and informational chunks follow.
• Output is ordered by authority and relevance.
• A strict 12K character limit prevents overwhelming the LLM with excessive context.

Integration: Making Axiom Accessible

A powerful backend is only as good as its accessibility. We designed Axiom with multiple integration points:

Internal Management with tRPC (src/server/trpc/routers/axiom.ts)

Our internal applications manage Axiom content via a comprehensive tRPC router, providing endpoints for:

• upload, confirmUpload, fetchUrl
• list, delete, update, reprocess
• search, stats
• A nested tokens router for API key management.

External Access with REST APIs (src/app/api/v1/rag/...)

For external systems or simpler integrations, we exposed key functionalities via REST endpoints:

• /api/v1/rag/search: For querying Axiom's knowledge base.
• /api/v1/rag/ingest: For programmatic document ingestion.
• /api/v1/rag/documents: For managing existing documents.
• Secure access is enforced with authenticateAxiomToken() middleware in src/app/api/v1/middleware.ts.

Workflow Engine Integration ({{axiom}} Variable)

The true power of Axiom comes from its direct integration into our workflow engine. By simply including {{axiom}} in a prompt or workflow step, the system automatically fetches and injects relevant context. This was implemented across:

• src/server/services/workflow-engine.ts (in resolvePrompt, runWorkflow, buildChainContext).
• src/lib/constants.ts for common step templates (e.g., "deep analysis", "synthesis").

Finally, a dedicated "Axiom" tab (with a Shield icon) has been added under the "Knowledge" group in our project detail page (src/app/(dashboard)/dashboard/projects/[id]/page.tsx), providing a user-friendly interface for managing documents.

Lessons Learned Along the Way

No development session is without its quirks. Here are a few "aha!" moments that saved the day:

• UI Component Variants: I initially tried using variant="outline" on some Badge components in the AxiomTab, only to be met with a TypeScript error. Turns out our Badge component only supports default, accent, success, warning, or danger. A quick switch to default or accent resolved it, reminding me to always check component props!
• pdf-parse v2 API Changes: Upgrading pdf-parse to v2.4.5 brought a new API. The old import pdfParse from "pdf-parse" no longer worked due to its class-based structure. The solution involved a dynamic import and instantiation:
  typescript

  const { PDFParse } = await import("pdf-parse");
  const parser = new PDFParse({ data: new Uint8Array(buffer) });
  const text = await parser.getText();
  parser.destroy(); // Important for memory management
  

  This highlights the importance of checking release notes for major version bumps!
• PostgreSQL Credentials: A classic, but always frustrating. Trying to run psql without proper credentials led to socket and password authentication errors. The fix was, as always, to properly reference the DATABASE_URL from our .env file: psql $DATABASE_URL -f prisma/rls.sql.

What's Next for Axiom?

With the core system in place, the immediate next steps involve:

1. Applying the final database schema changes and RLS policies.
2. Thorough end-to-end testing of the document upload, chunking, and search flows.
3. Expanding user-facing documentation for the {{axiom}} template variable in our workflow builder UI.
4. Looking ahead, we'll be exploring advanced features like "persona document scanning," where each AI persona can interpret and utilize documents tailored to its role.

Project Axiom represents a significant leap forward in empowering our applications with dynamic, contextual knowledge. It's exciting to see this vision come to life, and I'm eager to witness the intelligent workflows it will enable!