From Chat to Workflow: Unlocking Actionable Knowledge from Discussions with LLMs

In the fast-paced world of software development, valuable insights are often born in the crucible of team discussions – be it in chat, comments, or dedicated conversation threads. The challenge? These nuggets of wisdom often remain trapped within the discussion context, making it hard to bridge them directly into actionable workflows.

That's precisely the problem we set out to solve in our latest development sprint. Our goal: implement "Discussion Knowledge Export." We wanted to create a seamless pipeline to extract key insights from discussions and make them readily available as WorkflowInsight records, leveraging our existing MemoryPicker and {{memory}} pipeline for integration into automated workflows.

After a focused session, I'm thrilled to report that the feature is fully implemented and ready for testing! Let's pull back the curtain on how we built it.

The Core Idea: Bridging Conversations to Action

Imagine you've had a crucial design discussion that outlines a new feature's requirements, edge cases, and proposed solutions. Instead of manually distilling these points into a separate document or task, what if an AI could do it for you, making those insights immediately available when you're building a workflow related to that feature?

That's the power of Discussion Knowledge Export. By integrating LLMs, we can automatically digest lengthy discussions, extract core insights, assign a usefulness score, and store them in a structured format (WorkflowInsight). This means that when you're setting up a new workflow, relevant knowledge from past discussions is just a {{memory}} away.

Under the Hood: Building the Knowledge Engine

Bringing this vision to life required a multi-faceted approach, touching our data model, backend services, API, and frontend components.

Data Model Evolution

First, we needed to enrich our Discussion model to store the results of the extraction. We added three new nullable fields in prisma/schema.prisma:

• summary: A concise digest of the discussion.
• usefulnessScore: An LLM-generated score indicating the relevance/quality of the discussion's insights (e.g., how "actionable" it is).
• exportedAt: A timestamp to track when the knowledge was last exported.

A quick db:push and generate brought our database up to speed without needing a full migration, thanks to the nullable fields.

The Brain: LLM-Powered Extraction Service

The heart of the system lives in src/server/services/discussion-knowledge.ts. This service orchestrates the LLM interactions:

1. Parallel Haiku Calls: We leverage Anthropic's Haiku model for efficiency. Two parallel calls are made: one to generate the summary of the discussion, and another to extract specific WorkflowInsight records. This parallel execution minimizes latency.
2. Insight Persistence: The extracted insights, along with their embeddings, are then stored as WorkflowInsight records. These embeddings are crucial for semantic search and retrieval later in the MemoryPicker.
3. Tenant-Scoped Cleanup: To ensure data integrity and prevent duplication during re-exports, we implemented a tenant-scoped cleanup mechanism. When a discussion is re-exported, any old WorkflowInsight records linked to that discussion for the current tenant are removed before new ones are persisted.

API & Integration

Our src/server/trpc/routers/discussions.ts received three new tRPC procedures to expose this functionality:

• exportKnowledge (mutation): The primary endpoint to trigger the LLM-powered extraction. It includes LLM rate limiting for responsible usage.
• byProject (query): Allows fetching discussions linked to a specific project.
• getExportedInsights (query): Retrieves the WorkflowInsight records associated with a discussion.

Bringing it to Life: The User Experience

The frontend components were crucial for making this powerful backend accessible and intuitive:

• Usefulness Badge: src/components/discussion/usefulness-badge.tsx provides an immediate visual cue for the quality of an exported discussion. It's color-coded: gray (<30%), yellow (30-60%), green (60-80%), and our accent color (80%+) for highly useful discussions.
• Export Knowledge Dialog: src/components/discussion/export-knowledge-dialog.tsx is a sleek bottom sheet. From any discussion page, users can click "Export Knowledge," select a project to link the insights to, and initiate the extraction. A success state then displays the generated summary, usefulness score, and the number of insights extracted.
• Seamless Integration:
  • On individual discussion pages (discussions/[id]/page.tsx), an "Export Knowledge" button now sits in the header, and a summary bar appears below it once the discussion has been exported.
  • In the main discussions list (discussions/page.tsx), the UsefulnessBadge is visible on exported discussions, offering a quick overview.
  • Crucially, a new "Conversations" tab has been added to project detail pages (projects/[id]/page.tsx), providing a centralized view of all discussions linked to that project, complete with their scores and summaries.
  • Finally, the extracted discussion insights seamlessly appear in our MemoryPicker alongside workflow insights, ready to be selected and injected into new workflows using the {{memory}} syntax. This truly closes the loop, transforming raw conversations into actionable knowledge.

Throughout this process, security was paramount. Code review led to several fixes, including tenant-scoped raw SQL cleanup, robust Prisma JSON filtering using AND arrays, updateMany with tenantId validation, and project ownership validation for all mutations.

Challenges & Lessons Learned

No development sprint is without its bumps! Here are a couple of key lessons we picked up:

• The replace_all Gotcha: I initially tried to use a global replace_all to update all call sites for cleanupOldInsights. However, it only matched two out of three instances because one had different leading whitespace (2 spaces vs. 4 spaces indentation).

  • Lesson: Always double-check replace_all results, especially when dealing with code formatting variations. Manual verification is key for critical changes.

• Prisma's JSON Filtering Nuances: During code review, a potential issue was flagged with how I was constructing a Prisma JSON path filter, using metadata at two levels (where.metadata + AND.metadata). There was concern the second filter might shadow the first.

  • Lesson: When applying multiple conditions to the same field, especially with JSON filters, it's safer and more explicit to use Prisma's AND array syntax: AND: [{ metadata: ... }, { metadata: ... }]. This ensures all conditions are applied correctly and avoids unexpected behavior.

What's Next? Testing & Refinement

The feature is live on my dev server at http://localhost:3000, and the next immediate steps are all about rigorous testing:

1. Manual Test: Open a discussion with 5+ messages, click "Export Knowledge," and verify the summary, score, and insights appear correctly.
2. Re-export Test: Re-export the same discussion to ensure old insights are replaced, not duplicated.
3. Project Integration Test: Check the project detail "Conversations" tab to confirm linked discussions with scores are displayed.
4. MemoryPicker Integration Test: Create a new workflow, open the MemoryPicker, and verify discussion insights appear alongside workflow insights.
5. Workflow Execution Test: Run a workflow with selected discussion insights and confirm {{memory}} correctly contains the discussion knowledge.
6. Future Consideration: The byProject query currently returns all discussions. We'll consider adding pagination to this query in the near future for performance with larger datasets, as flagged during review.

Conclusion

This feature is a significant step forward in making our platform more intelligent and productive. By transforming unstructured discussions into actionable, retrievable knowledge, we're empowering users to build more robust and informed workflows. It's exciting to see how LLMs can bridge the gap between human conversation and automated action, truly enhancing the developer experience.

---