From CRUD List to Knowledge Hub: Unifying Data and Intelligence in a Next.js App

It started, as many things do, with a single page: /dashboard/memory. For months, it had served as a simple, albeit growing, CRUD list. A timeline of MemoryEntry objects, each a snippet of thought, a recorded observation. Functional, yes. Intelligent? Far from it. At 194 lines and counting, it was becoming a bottleneck, a data graveyard where valuable insights went to languish without context or connection.

Our vision was ambitious: transform this humble list into a living, breathing Knowledge Hub. A place where not just MemoryEntry data, but also WorkflowInsight, ConsolidationPattern, and CodePattern could converge. The goal was clear: unify these four disparate data sources, present them in clustered views, offer smart suggestions, and enable powerful vector-based semantic search.

This post chronicles the journey from a simple list to a sophisticated intelligence layer, detailing the architectural decisions, the implementation steps, and a critical lesson learned about sharing logic between client and server in Next.js.

The Core Problem: Data Silos and Lack of Context

Our application generated a lot of valuable data across different domains:

• MemoryEntry: Raw observations, notes, quick thoughts.
• WorkflowInsight: Higher-level observations about processes and efficiency.
• ConsolidationPattern: Identified reusable solutions or approaches.
• CodePattern: Specific code snippets or architectural patterns.

Each of these lived in its own corner, accessible via its own specific route. The /dashboard/memory page was meant to be a central point, but it only showed one type of data. Finding connections, identifying overarching themes, or leveraging past solutions required manual effort and a good memory. We needed a system that could do the heavy lifting for us.

Building the Brain: Backend Services

The heart of our Knowledge Hub resides in src/server/services/knowledge-hub.ts. This service is responsible for orchestrating the unification and intelligence layers.

1. Data Normalization: The first step to unification was ensuring all four data types could be treated similarly. We created four dedicated normalizer functions (e.g., normalizeMemoryEntry, normalizeWorkflowInsight) that transform each source's specific schema into a common KnowledgeItem structure. This KnowledgeItem type, defined in src/types/knowledge.ts, became our universal currency.

2. Unified Querying (queryKnowledgeHub()): This function now acts as the single entry point for fetching all relevant knowledge. It queries each of the four underlying data sources, normalizes their results, and then aggregates them.

3. Algorithmic Suggestions (generateSuggestions()): While we plan for LLM integration down the line, our initial suggestion engine is purely algorithmic. It identifies pain points (e.g., from MemoryEntry items tagged as "problem") and attempts to pair them with known solutions or patterns (e.g., ConsolidationPattern items). This allows us to generate Workaround, Prompt, or Best Practice suggestions based on existing data.

4. Semantic Search (findRelatedKnowledge()): Leveraging our existing pgvector setup, findRelatedKnowledge() allows us to perform semantic searches, finding items that are conceptually similar even if they don't share keywords. This is a game-changer for discovering hidden connections.

All these powerful backend capabilities were then exposed through new tRPC procedures in src/server/trpc/routers/memory.ts: knowledgeHub, knowledgeStats, suggestions, and relatedKnowledge.

Bringing it to Life: Frontend Components

The user experience is paramount for a Knowledge Hub. We built a dedicated suite of 11 components within src/components/knowledge/ to render this rich data:

• knowledge-search.tsx: A debounced search input, complete with a "Sparkles" icon toggle for semantic (vector) search, allowing users to switch between keyword and conceptual queries.
• knowledge-filter-bar.tsx: A dynamic set of chip filters for source, cluster, severity, and project, enabling granular data exploration.
• knowledge-item-card.tsx: An expandable card for each KnowledgeItem, displaying source, severity, tags, and a crucial "Find related" button to trigger contextual semantic searches.
• knowledge-cluster-group.tsx: Collapsible sections that organize KnowledgeItems into logical clusters like "Issues," "Solutions & Patterns," "Tools," and "Notes."
• knowledge-stats.tsx: Four visually appealing stat cards (Knowledge Assets, Solutions & Patterns, Resolution Rate, Intelligence Coverage) provide an at-a-glance overview, framed positively to highlight progress and value.
• knowledge-category-chart.tsx & severity-heatmap.tsx: Visualizations using Recharts to give users a quick understanding of their knowledge distribution and common severity levels.
• suggestion-card.tsx & knowledge-suggestions.tsx: Dedicated components to display and orchestrate the algorithmic suggestions, complete with confidence bars and copy-to-clipboard functionality.
• related-knowledge.tsx: A panel to display vector-similar items, providing immediate context and discovery.

Finally, the entire experience was orchestrated within src/app/(dashboard)/dashboard/memory/page.tsx. We opted for a 3-tab layout: Knowledge, Suggestions, and Entries. Crucially, the "Entries" tab preserves the original CRUD timeline, ensuring that users can still access the raw data in its familiar format while introducing the new, intelligent views.

⚠️ A Critical Lesson Learned: Client/Server Boundaries ⚠️

During development, we hit a classic Next.js pitfall that's worth highlighting.

• The Problem: We needed a consistent way to map KnowledgeItems to their display clusters (e.g., "Issues", "Solutions"). This logic, getCluster(), was initially placed in src/server/services/knowledge-hub.ts because it felt like a core backend concern.
• The Attempt: On the client-side page.tsx (a "use client" component), we tried to import getCluster() directly from the server service to apply the same clustering logic for frontend display.
• The Failure: This immediately caused a bundling error. Why? Because knowledge-hub.ts imports PrismaClient, which is a server-only dependency. The Next.js bundler, when trying to include knowledge-hub.ts in the client bundle, choked on the PrismaClient import.
• The Solution: The getCluster() function, along with its associated CLUSTER_MAP, was a pure function – it didn't interact with the database or any other server-specific resources. We extracted it to src/types/knowledge.ts. This directory is explicitly for shared types and pure utility functions that have no server-side dependencies. Both the server service and the client page could then safely import getCluster() from this neutral ground.

// src/types/knowledge.ts (simplified)

export type KnowledgeCluster = "issues" | "solutions" | "patterns" | "tools" | "notes";

export const CLUSTER_MAP: Record<string, KnowledgeCluster> = {
  // ... mappings for different item types/tags to clusters
  'MemoryEntry-problem': 'issues',
  'WorkflowInsight-optimization': 'solutions',
  'ConsolidationPattern': 'patterns',
  'CodePattern': 'tools',
  // ...
};

export function getCluster(item: KnowledgeItem): KnowledgeCluster {
  // Pure logic to determine cluster based on item's type, tags, etc.
  // Example:
  if (item.source === 'MemoryEntry' && item.tags.includes('problem')) return 'issues';
  if (item.source === 'ConsolidationPattern') return 'patterns';
  // ... more complex logic
  return 'notes'; // Default
}

Takeaway: Any logic intended for use by both client components and server services must be completely independent of server-side dependencies (like database clients, file system access, etc.). src/types/ or src/lib/ are ideal homes for such pure, shared utilities. Ignoring this boundary leads to frustrating bundling errors and bloated client bundles.

What's Next?

The Knowledge Hub is feature-complete and pushed to origin/main. Our immediate next steps involve rigorous testing and refinement:

1. Data Verification: Ensuring all four sources render and cluster correctly with real-world data.
2. Semantic Search Fallback: Verifying the semantic search toggle gracefully falls back to keyword search if an OpenAI key isn't configured.
3. Suggestion Accuracy: Confirming the pain-solution pairing generates relevant workaround cards.
4. Scalability: Considering pagination controls, as currently, we cap at 100 items.
5. Responsiveness: Polishing the mobile layout for stacked filters and single-column clusters.
6. Pre-existing Bug Fix: Addressing an unrelated Suspense boundary build error on /dashboard/consolidation/new.

This transformation from a simple CRUD list to an intelligent Knowledge Hub has been a significant step in making our application more powerful and our data more actionable. It's a testament to how thoughtful architecture and a focus on user experience can unlock hidden value from existing data.

{
  "thingsDone": [
    "Unified 4 distinct data sources (MemoryEntry, WorkflowInsight, ConsolidationPattern, CodePattern) into a single Knowledge Hub view.",
    "Implemented backend services for data normalization, unified querying, algorithmic suggestions, and pgvector-based semantic search.",
    "Developed 11 dedicated frontend components for search, filtering, display, statistics, and intelligent suggestions/related items.",
    "Refactored the dashboard page into a 3-tab layout, preserving the original CRUD view while introducing new intelligence layers.",
    "Applied positive framing and visual polish to statistics and empty states."
  ],
  "pains": [
    "Encountered bundling error when importing server-only code (PrismaClient) into a client-side component via a shared service."
  ],
  "successes": [
    "Successfully extracted pure, shared logic (getCluster, CLUSTER_MAP) to a neutral, dependency-free location (src/types/knowledge.ts), resolving the bundling issue.",
    "Achieved the goal of transforming a basic CRUD list into an intelligent, unified Knowledge Hub.",
    "Integrated pgvector for semantic search capabilities, enhancing data discovery."
  ],
  "techStack": [
    "Next.js",
    "tRPC",
    "Prisma",
    "PostgreSQL (with pgvector)",
    "TypeScript",
    "Recharts",
    "TailwindCSS (implied by component structure)"
  ]
}