← nyxcore-systems
6 min read

Beyond Project Silos: An AI-Powered Journey to Consolidate Lessons Learned

Dive into the journey of building an AI-powered feature to consolidate insights across multiple projects, tackling context limits, Prisma types, and even AI agent limitations along the way.

AILLMFullstackPrismaTypeScriptNext.jsSoftware DevelopmentKnowledge Management

In the fast-paced world of software development, lessons are learned daily. New solutions emerge, old pains resurface, and innovative tools are adopted. But how often do these valuable insights remain locked within individual project documentation, lost to future endeavors, or buried in an ever-growing backlog of notes?

That was the challenge we set out to solve: to move beyond project silos and create a system that intelligently consolidates our collective wisdom. The goal was ambitious: build a cross-project consolidation feature that could accumulate letters, blog posts, and other project memories, extract patterns (successes, pains, solutions, tools, architecture) via AI, provide a searchable overview, and even export these patterns as prompt hints for future use.

After an intense development sprint, I'm thrilled to report that this feature is now complete, tested end-to-end, and pushed to origin/main. Let's unpack the journey.

The Vision: Unlocking Collective Wisdom

Imagine an intelligent assistant that sifts through all your project documentation, identifying recurring themes, pinpointing common pitfalls, and highlighting successful strategies. This isn't just about search; it's about synthesis. Our new consolidation feature aims to:

  • Aggregate: Pull in relevant project memories from multiple sources.
  • Analyze with AI: Use large language models to identify patterns across projects – what worked, what didn't, what tools were effective.
  • Organize & Search: Provide a structured, searchable overview of these extracted patterns.
  • Empower: Export these patterns as actionable prompt hints, ready to guide new projects or inform strategic decisions.
  • Integrate: Make these insights accessible directly from individual project detail pages.

The core of this feature lies in its ability to connect disparate pieces of information, creating a cohesive narrative of our development journey across the entire organization.

Bringing the Vision to Life: The Development Breakdown

Building this required a full-stack approach, touching database schema, backend services, API routes, and a comprehensive user interface.

Backend Foundations: Data, Logic, and APIs

Our journey began by laying down the architectural groundwork:

  • Data Models (prisma/schema.prisma): We introduced Consolidation and ConsolidationPattern models. The Consolidation model stores references to the projectIds it encompasses (String[] @db.Uuid), ensuring a clear link to the original projects. Cascade deletes and robust tenant/user relations were, of course, a given.
  • AI Pattern Extraction (src/server/services/consolidation-service.ts): This is the brain of the operation.
    • extractPatterns(): Orchestrates the AI call, requesting structured JSON output for identified patterns (successes, pains, solutions, etc.).
    • generatePromptHints(): Transforms these patterns into categorized markdown, perfect for quick reference.
    • budgetContents(): A critical utility (more on this in the challenges section!) for intelligently truncating input to fit LLM context windows.
  • API Endpoints (src/server/trpc/routers/consolidation.ts): A comprehensive tRPC router was built to handle all aspects of consolidation management:
    • list, get, create, delete: Standard CRUD operations.
    • generate, regenerate: Triggering the AI analysis.
    • patterns.search: A paginated search endpoint with ILIKE for pattern discovery.
    • export: Providing markdown, JSON, or prompt-hint formats.
    • availableProjects, byProject: Helpers for project selection and integration.

Frontend Experience: Intuitive Interaction

With a robust backend, we focused on delivering a seamless user experience:

  • Navigation (src/components/layout/sidebar.tsx): A new "Consolidation" nav item with a clear Layers icon was added, making the feature easily discoverable.
  • Consolidation List Page (src/app/(dashboard)/dashboard/consolidation/page.tsx): Users can see an overview of all their consolidations, complete with status badges and pattern type counts for quick insights.
  • New Consolidation Form (src/app/(dashboard)/dashboard/consolidation/new/page.tsx): A guided experience allows users to select projects using a multi-selector with checkboxes and project counts. A step machine (idle → creating → analyzing → done) provides clear feedback during the AI processing phase. We also added URL parameter pre-selection for convenience.
  • Consolidation Detail Page (src/app/(dashboard)/dashboard/consolidation/[id]/page.tsx): This is where the magic happens. It features three tabs:
    • Overview: A summary and distribution bars for pattern types.
    • Patterns: A searchable list of extracted patterns with type filter chips and expandable cards for detail.
    • Export: Options to copy to clipboard or download patterns in various formats, with an inline preview.
  • Project Integration (src/app/(dashboard)/dashboard/projects/[id]/page.tsx): A new "Analysis" tab was added to individual project pages, showing relevant consolidations and a direct "Analyze Patterns" CTA to kick off a new consolidation with that project pre-selected.

Interestingly, for the UI development, we experimented with spawning specialist AI agents for different pages (list, new, detail, project tab). While agents handled simpler pages like the list view and project tab quickly, the more complex, multi-tabbed detail and new pages ultimately required manual intervention due to their intricate logic and interdependencies. A valuable lesson in itself!

Navigating the Treacherous Waters: Challenges & Solutions

No significant feature ships without its share of hurdles. Here are some of the critical challenges we faced and how we overcame them:

1. The LLM Context Window Wall

  • The Problem: Our initial approach was to send all project memory entries and blog posts (around 10 letters and 9 blog posts for a typical consolidation) directly to the Anthropic API. This quickly hit a wall: Anthropic API error: 400 input length and max_tokens exceed context limit: 198654 + 8192 > 200000. LLM context windows, while generous, are not infinite.
  • The Solution: We implemented budgetContents() within our consolidation-service.ts. This function intelligently truncates individual entries to a maximum of 15,000 characters and caps the total input for the LLM at 500,000 characters. Crucially, we also realized that blog posts are often derived from letters and contain redundant information for pattern extraction. Removing them from the input significantly reduced token usage without losing valuable insights.
  • Lesson Learned: LLM context limits are a fundamental constraint. Strategic content selection and intelligent truncation are vital for efficient and cost-effective AI interactions. Don't just throw everything at the model; curate your input.

2. Taming Prisma's Json? Types in TypeScript

  • The Problem: When working with Prisma's Json? fields (nullable JSON), we encountered TypeScript errors trying to assign Record<string, unknown> types: TS2322: Type 'Record<string, unknown>' is not assignable to type 'NullableJsonNullValueInput | InputJsonValue'.
  • The Solution: The key was to explicitly import Prisma from @prisma/client and cast our objects to Prisma.InputJsonValue for valid JSON values, and use Prisma.JsonNull when we explicitly wanted to set the field to null in the database.
  • Lesson Learned: Prisma's type definitions for JSON fields, especially with nullability, require specific handling. Leveraging the Prisma namespace directly provides the necessary type safety and correctness.

3. When AI Agents Hit Their Limits

  • The Problem: As mentioned earlier, while our specialist AI agents performed well on simpler, focused UI components, they struggled with the complexity of multi-tabbed pages like the consolidation detail view or the multi-step new consolidation form. They would often get stuck "in_progress" for extended periods (10+ minutes) without generating files.
  • The Solution: For these intricate pages, we took over and wrote the code manually.
  • Lesson Learned: AI agents are powerful tools for accelerating development on well-defined, isolated tasks. However, for complex UI flows, nuanced state management, and intricate integration logic, human developers remain indispensable. It's about finding the right balance between automation and manual craftsmanship.

The Road Ahead

With the core feature solid and tested, our immediate next steps involve:

  • Thorough end-to-end verification of the full consolidation lifecycle: create → generate → view patterns → export prompt hints.
  • Testing the regenerate flow from the detail page.
  • Ensuring robust mobile layout and clipboard functionality.
  • Considering a global pattern search across all consolidations for even broader insights.

This consolidation feature represents a significant step forward in how we learn from our past projects. By harnessing AI, we're not just storing information; we're transforming it into actionable intelligence that can guide our future development.