← nyxcore-systems
6 min read

From CRUD List to Knowledge Hub: Unifying Insights with Next.js and tRPC

We transformed a simple CRUD list into a powerful Knowledge Hub, unifying disparate data sources, implementing smart suggestions, and leveraging vector search for deeper insights. Here's how we built it and the lessons we learned.

Next.jstRPCTypeScriptFull-StackKnowledge ManagementVector SearchUI/UXPrismapgvector

Every developer knows the feeling: you have a simple list, a functional CRUD interface. It does its job, but it's just... a list. What if that list held the keys to deeper insights, connecting disparate pieces of information across your application? What if it could not just show data, but suggest solutions, highlight patterns, and reveal relationships?

That was the challenge we faced with our /dashboard/memory page. What started as a straightforward 194-line component for managing "Memory Entries" had the potential to become something far greater: a central Knowledge Hub. Our goal was ambitious: unify four distinct data sources—MemoryEntry, WorkflowInsight, ConsolidationPattern, and CodePattern—into a cohesive, intelligent interface complete with clustered views, smart suggestions, and semantic search.

This post chronicles the journey of transforming that humble list into a powerful, insightful knowledge repository.

The Vision: A Unified Brain for Our Application

Imagine having all your project's critical information – individual notes, workflow insights, common consolidation patterns, and even code patterns – living in separate silos. The /dashboard/memory page was originally just one of those silos. Our vision for the Knowledge Hub was to break down these barriers, creating a single source of truth where:

  1. All Knowledge is Centralized: A unified view of MemoryEntry, WorkflowInsight, ConsolidationPattern, and CodePattern.
  2. Patterns Emerge: Data isn't just listed; it's clustered into logical groups like "issues," "solutions," "patterns," "tools," and "notes."
  3. Intelligence Guides You: Algorithmic suggestions offer workarounds and best practices, while vector search (pgvector) finds semantically related items, even if keywords don't match.
  4. Insights are Actionable: Key statistics provide a high-level overview of our collective knowledge and its impact.

Building the Brain: Our Technical Journey

Bringing this vision to life required a full-stack effort, touching data modeling, backend services, API design, and a complete UI overhaul.

1. Modeling the Unified Knowledge

The first step was to define a common language for our diverse data. We introduced src/types/knowledge.ts, which became home to our core data structures:

  • KnowledgeItem: A normalized representation of any of the four data sources, allowing us to treat them uniformly in the UI and backend services.
  • KnowledgeSuggestion: The structure for our intelligent recommendations.
  • KnowledgeStatsData: For presenting high-level metrics.
  • Crucially, this file also housed a shared getCluster() mapping function and CLUSTER_MAP (more on why this was important later!).

2. The Backend Engine: Services & tRPC Procedures

The heavy lifting on the server side was handled by src/server/services/knowledge-hub.ts. This service orchestrates the fetching, normalizing, and processing of all knowledge.

  • Normalization: We created four dedicated normalizer functions (e.g., normalizeMemoryEntryToKnowledgeItem) to convert each source's specific schema into our generic KnowledgeItem type. This was fundamental for unification.
  • Unified Querying: The queryKnowledgeHub() function became the single entry point for fetching data, abstracting away the complexity of querying multiple tables.
  • Intelligent Features:
    • generateSuggestions(): This function implements our algorithmic suggestion logic, pairing common "pains" with recorded "solutions" or "workarounds." Critically, this is algorithmic, not relying on an external LLM, demonstrating smart internal logic.
    • findRelatedKnowledge(): We leveraged our existing pgvector setup to perform semantic searches, finding items related by meaning rather than just keywords.
    • getKnowledgeStats(): Aggregates metrics for our dashboard.

To expose these capabilities to the frontend, we extended src/server/trpc/routers/memory.ts with four new tRPC procedures: knowledgeHub, knowledgeStats, suggestions, and relatedKnowledge. This provided a type-safe, efficient API for our client.

3. Crafting the User Experience: A New Frontend

The frontend transformation was the most visible change. We completely rewrote src/app/(dashboard)/dashboard/memory/page.tsx into a sophisticated 3-tab layout:

  • Knowledge Tab: The main hub, featuring:

    • KnowledgeSearch: A debounced search bar with a semantic search toggle (a sparkling icon!).
    • KnowledgeFilterBar: Chip-based filters for source, cluster, severity, and project.
    • KnowledgeClusterGroup: Collapsible sections for each cluster (issues, solutions, patterns, tools, notes), making it easy to navigate large datasets.
    • KnowledgeItemCard: Expandable cards for each knowledge item, showing source, severity, tags, evidence, and a "Find related" button.
    • KnowledgeStats and KnowledgeCategoryChart: Visually appealing cards and a Recharts bar chart to summarize our knowledge assets, solutions, resolution rates, and intelligence coverage (framed positively!).
    • SeverityHeatmap: A proportional, color-coded strip to visualize severity distribution.

  • Suggestions Tab: Powered by knowledge-suggestions.tsx, this tab presents SuggestionCard components, offering prompt workarounds and best practices with a confidence bar and copy button.

  • Entries Tab: We made sure to preserve the original CRUD timeline functionality here, ensuring continuity for users who still needed the raw list view.

The UI was polished with positive framing for stats, color-coded values, and aspirational empty states, ensuring a delightful user experience even when data is sparse.

Overcoming Hurdles: A Lesson in Full-Stack Boundaries

No significant development journey is without its challenges. One particular hurdle taught us a valuable lesson about the client-server boundary in a Next.js application using tRPC:

The Problem: We initially tried to import getCluster() from src/server/services/knowledge-hub.ts directly into our "use client" page component. The getCluster() function itself was pure, but knowledge-hub.ts also imported PrismaClient—a server-only dependency.

The Failure: This caused a bundling error. Next.js's client-side bundle tried to include PrismaClient, which is designed to run only on the server, leading to a build failure.

The Lesson & Workaround: The critical insight here is that any shared logic between client components and server services must be completely pure and free of server-only dependencies.

Our solution was to extract getCluster() and CLUSTER_MAP into src/types/knowledge.ts. Since src/types/ is explicitly for shared type definitions and utility functions that are pure, it contains no server-specific imports like PrismaClient. Both our server service (knowledge-hub.ts) and our client page (page.tsx) could then safely import getCluster() from this shared, dependency-free location. This reinforces the importance of careful dependency management across the client-server divide.

Current State and Next Steps

We're thrilled to announce that the Knowledge Hub is feature-complete! The core logic is committed (39bb3fe, 9b5e7a8) and pushed to origin/main. The dev server is running smoothly on port 3000, ready for testing.

Our immediate next steps involve rigorous testing:

  1. Data Validation: Ensuring all four knowledge sources render and cluster correctly with real data.
  2. Semantic Search: Verifying the semantic search toggle works as expected and gracefully falls back if an OpenAI key isn't present.
  3. Suggestions: Confirming the algorithmic pain-solution pairing generates relevant workaround cards.
  4. Refinements: Considering pagination for larger datasets and polishing the mobile layout for a responsive experience.
  5. Squashing Bugs: Addressing a pre-existing build error on /dashboard/consolidation/new (unrelated to this work, but important for overall stability).

Conclusion

Transforming a simple list into an intelligent Knowledge Hub has been a challenging yet incredibly rewarding journey. By unifying disparate data, implementing smart suggestion algorithms, leveraging vector search, and carefully designing the user experience, we've created a powerful tool that doesn't just display information, but helps users discover insights and make better decisions. This project truly highlights the power of a modern full-stack approach with Next.js, tRPC, and a keen eye for architectural boundaries. We're excited to see how this hub evolves and empowers our users!