Project Axiom: Shipping a Full-Stack RAG System in One Epic Sprint

The feeling of a major development sprint wrapping up is always a unique blend of exhaustion and exhilaration. Today, that feeling is particularly strong. We just pushed Project Axiom – our ambitious, full-stack RAG (Retrieval-Augmented Generation) knowledge system – from concept to a fully implemented, type-checked, and commit-ready state. Sixteen new files, countless lines of code, and a significant leap forward in how our system leverages internal knowledge.

Project Axiom is designed to be the 'knowledge brain' of our platform, allowing users to upload documents, have them intelligently processed, and then use that contextual information to enhance LLM-driven workflows. Think of it as a sophisticated digital librarian, ready to provide precise, relevant context on demand.

The Mission: Building Axiom's Knowledge Core

Our goal was clear: implement a robust RAG system that could handle document ingestion, intelligent chunking, embedding, hybrid search, and seamless integration into our existing workflow engine. This wasn't just about wiring up an LLM; it was about building the entire scaffolding that makes an LLM truly useful with proprietary data.

Here's how we broke it down and what we achieved:

1. The Database Foundation: PostgreSQL as Our Knowledge Vault

At the heart of any RAG system is robust data storage. We leveraged our existing PostgreSQL database, extending it significantly with Prisma to handle the unique demands of RAG:

• Prisma Models: We introduced three core models in prisma/schema.prisma:
  • ProjectDocument: For managing the uploaded documents themselves, including metadata like authority and category to influence search relevance.
  • DocumentChunk: The granular units of knowledge, each storing its vector embedding (for semantic search) and tsvector (for full-text search).
  • AxiomApiToken: For secure external access to the Axiom API, storing SHA-256 hashed tokens with expiry and usage tracking.
• PostgreSQL Power-Up: This is where things got exciting. We didn't just store data; we made it searchable and secure:
  • Row-Level Security (RLS): Implemented in prisma/rls.sql to ensure documents are only accessible by authorized users/projects.
  • Vector Column & HNSW Index: For efficient similarity search on our document embeddings.
  • GIN FTS Index & tsvector Trigger: To power fast, linguistic full-text search and automatically update our search index on document changes.

-- Example snippet from prisma/rls.sql (conceptual)
ALTER TABLE "DocumentChunk" ADD COLUMN "embedding" vector(1536);
CREATE INDEX ON "DocumentChunk" USING HNSW ("embedding" vector_cosine_ops);
CREATE INDEX ON "DocumentChunk" USING GIN ("content_tsv");

This approach allowed us to use PostgreSQL as a powerful hybrid vector/relational database, simplifying our infrastructure by avoiding an additional dedicated vector store.

2. Intelligent Document Processing: The Art of Chunking

Raw documents are rarely useful for RAG; they need to be broken down into digestible chunks. Our src/server/services/rag/document-processor.ts became the brain for this:

• Strategy-based Chunking: We implemented different chunking strategies tailored to content types:
  • Markdown: By headings, preserving logical structure.
  • Plain Text: By paragraphs, maintaining semantic units.
  • Code: By declarations, ensuring code snippets remain intact.
  • PDF: Via pdf-parse v2, splitting by pages to manage context.
• Storage Adapter Extension: Our src/server/services/storage.ts was updated to handle various document MIME types, with a sensible 50MB max limit to prevent system overload.

3. Hybrid Search & Relevance Tuning: Finding the Needle in the Haystack

Retrieving the right context is paramount. Our src/server/services/rag/document-search.ts implements a sophisticated hybrid search mechanism:

• 70% Vector + 30% FTS: This blend ensures we capture both semantic similarity (vector) and exact keyword matches (full-text), providing a more robust retrieval.
• Authority-Level Score Boosting: Crucially, we introduced a mechanism to boost relevance based on document authority (mandatory +0.3, guideline +0.15). This ensures critical information is prioritized.
• Content Loading Strategy: src/server/services/rag/load-axiom-content.ts orchestrates the final content assembly: mandatory chunks first, then relevant guideline/informational, all ordered by authority, capped at a 12K character limit for prompt efficiency.

4. Secure & Integrated APIs: Accessing Axiom's Power

For both internal and external consumption, Axiom needed robust API surfaces:

• tRPC Router: src/server/trpc/routers/axiom.ts provides a full internal API for managing Axiom: upload, confirm, fetch URL, list, delete, update, reprocess documents, search, and even token management. This was registered in src/server/trpc/router.ts.
• REST Endpoints: For external integrations, we exposed key functionalities via src/app/api/v1/rag/search/route.ts, src/app/api/v1/rag/ingest/route.ts, and src/app/api/v1/rag/documents/route.ts.
• API Token Security: src/server/services/rag/token-service.ts handles nyx_ax_ prefixed API tokens, SHA-256 hashing, expiry checks, and lastUsedAt tracking, secured by authenticateAxiomToken() middleware.

5. Bringing it to Life: Workflow & UI Integration

The true power of Axiom comes from its integration into our existing platform:

• {{axiom}} Template Variable: This is a game-changer. By integrating {{axiom}} into src/server/services/workflow-engine.ts (specifically resolvePrompt, runWorkflow, buildChainContext), users can now dynamically inject relevant context from Axiom directly into their LLM prompts within workflows. We even added it to our src/lib/constants.ts for deep analysis and synthesis steps.
• User Interface: A new "Axiom" tab (with a distinctive Shield icon) was added under the "Knowledge" group in the project detail page (src/app/(dashboard)/dashboard/projects/[id]/page.tsx), providing a user-friendly interface for managing documents.

Lessons Learned from the Trenches (The "Pain Log" Transformed)

Not everything was smooth sailing. Here are a few valuable lessons from the snags we hit:

1. UI Component Quirks: Always Check the Docs (or Types!)

   • The Problem: I tried to use variant="outline" on some Badge components in the AxiomTab, expecting a common UI pattern. TypeScript quickly informed me that the component only supported "default", "accent", "success", "warning", "danger".
   • The Lesson: While TypeScript is a fantastic guardrail, it's a good reminder to quickly consult component documentation or the component's type definitions directly when working with new UI libraries or components. It saves time trying to force a non-existent prop. My workaround was simply switching to default or accent, which worked fine visually.

2. Dependency Migrations: pdf-parse v1 vs v2 - A Tale of Two APIs

   • The Problem: Initially, I tried import pdfParse from "pdf-parse", which is the common way to import default exports. However, after upgrading to pdf-parse v2.4.5, it failed. The new version adopted a class-based API without a default export.
   • The Lesson: Major version bumps in dependencies often come with breaking API changes. Always check the changelog or migration guide. My workaround involved a dynamic import and class instantiation: const { PDFParse } = await import("pdf-parse"); new PDFParse({ data: new Uint8Array(buffer) }).getText().destroy();. This ensured compatibility and proper resource cleanup.

3. Environment Setup: The Unsung Hero - Your .env File

   • The Problem: When trying to run psql directly to apply RLS policies, I initially forgot to specify credentials, leading to socket and password authentication errors.
   • The Lesson: Database connection strings (and all sensitive environment variables) are critical. Always ensure your .env file is correctly configured and that tools like psql are invoked with the appropriate DATABASE_URL (e.g., psql $DATABASE_URL -f prisma/rls.sql). It's a basic, but often overlooked, step in local development.

Looking Ahead: The Road to Production & Beyond

With the core implementation complete, the immediate next steps are focused on deployment and validation:

1. Schema & Indexes: Apply the database changes (npm run db:push && npm run db:generate) and then the crucial RLS, vector, and FTS indexes/triggers (psql $DATABASE_URL -f prisma/rls.sql).
2. End-to-End Testing: Thoroughly test the document upload flow (Markdown, PDF), confirmation, chunk creation, and then hybrid search via both tRPC and the REST API.
3. Future Enhancements:
   • Persona Document Scanning: A user suggested "every persona scans document and interprets it." This is a fascinating idea but requires significant workflow trigger infrastructure – a definite deferred item for a future sprint.
   • UI Hints for {{axiom}}: Adding user-facing documentation and hints within the workflow builder UI will be critical for adoption of the new template variable.

Conclusion

Project Axiom represents a significant architectural evolution for our platform. We've built a powerful, flexible, and secure RAG system that integrates deeply into our existing workflows, unlocking new possibilities for intelligent automation and knowledge leverage. The journey was challenging, but the satisfaction of seeing all 16 files snap into place, type-checking cleanly, makes it all worthwhile. Here's to the next phase of bringing Axiom's knowledge to life!

{
  "thingsDone": [
    "Implemented 3 Prisma models: ProjectDocument, DocumentChunk, AxiomApiToken",
    "Added RLS policies, vector column, HNSW index, GIN FTS index, tsvector trigger in PostgreSQL",
    "Created document-processor.ts with chunking strategies for markdown, plain text, code, and PDF (via pdf-parse v2)",
    "Developed document-search.ts for hybrid search (70% vector + 30% FTS) with authority-level score boosting",
    "Built token-service.ts for secure API token management (SHA-256, expiry, lastUsedAt)",
    "Implemented load-axiom-content.ts for ordered, relevance-boosted content loading with character limits",
    "Created comprehensive tRPC router (axiom.ts) for RAG management",
    "Registered axiom router in main tRPC router",
    "Added authenticateAxiomToken() middleware for REST API security",
    "Created REST endpoints for RAG search, ingest, and document management",
    "Integrated {{axiom}} template variable into workflow-engine.ts for dynamic context injection",
    "Added 'Axiom' tab to project detail page UI",
    "Included {{axiom}} in step templates (deep analysis, synthesis)",
    "Extended storage adapter for document MIME types and max size",
    "Updated CLAUDE.md with documentation for new features",
    "Added pdf-parse v2.4.5 dependency"
  ],
  "pains": [
    "Badge component variant='outline' not supported (TypeScript error), replaced with 'default'/'accent'",
    "pdf-parse v1 vs v2 API incompatibility (class-based API), required dynamic import and specific instantiation",
    "psql credential issues, resolved by using DATABASE_URL from .env"
  ],
  "successes": [
    "All 16 files implemented and type-checking cleanly",
    "Robust RAG system with hybrid search and relevance boosting",
    "Seamless integration into existing workflow engine via template variable",
    "Secure API access with token management",
    "Leveraging PostgreSQL for both vector and full-text search, reducing infrastructure complexity"
  ],
  "techStack": [
    "TypeScript",
    "Prisma",
    "PostgreSQL (with HNSW extension)",
    "tRPC",
    "Next.js (API Routes)",
    "pdf-parse v2",
    "Vector Embeddings",
    "Full-Text Search (FTS)"
  ]
}