Unlocking Workflow Memory: How We Built an Intelligent Context System for Our AI Agents

It was late, the kind of late where the lines of code start to blur but the excitement of a breakthrough keeps you going. We were grappling with a fundamental challenge in our AI-driven workflow system: how do we empower our agents to truly learn from their past endeavors? Workflows were producing valuable outputs, but that knowledge often remained transient, locked within a single execution. We needed a way for key insights to persist, become searchable, and ultimately, inform future decisions.

Our goal? To implement a robust project-workflow memory system. Imagine an AI agent reviewing a completed task, identifying crucial "aha!" moments or critical failures, and then saving those as structured WorkflowInsight records. These insights would then be available via vector search, selectable as a context source, and seamlessly injected into new workflow creations using a simple {{memory}} template variable. This isn't just about storing data; it's about building a collective intelligence for our agentic workflows.

The Agentic Design Sprint: A Team of Three

To tackle this, we spun up a specialized agent team: an Architect, an ML Expert, and a UX Developer. This collaborative approach allowed us to address all facets of the problem simultaneously.

1. The Architect's Blueprint: Unearthing the Gaps

Our Architect kicked things off by mapping out our existing database models, tRPC routes, and service patterns. This initial research was critical and revealed some key limitations:

• No Project-Specific Context: Our MemoryEntry model lacked a projectId foreign key, meaning general memory was decoupled from specific project contexts.
• Underutilized Search: The searchVector (a tsvector column) was never populated, leaving full-text search capabilities dormant.
• Transient Knowledge: "Key points" were checkpoint-only and never consolidated or written back into a persistent memory store.
• One-Way Street: While workflows could read from memory, there was no mechanism for them to write back, creating a knowledge sink.

This foundational analysis confirmed our hypothesis: we needed a dedicated, project-aware memory system.

2. The ML Expert's Vision: Crafting the WorkflowInsight Model

Armed with the Architect's findings, our ML Expert designed the heart of our new system: the WorkflowInsight model. This wasn't just an extension of MemoryEntry; it was a distinct entity tailored for persistent, searchable knowledge.

Key features of the WorkflowInsight model:

• Dedicated Model: A clean separation from general MemoryEntry for clarity and purpose.
• pgvector Integration: To enable powerful semantic search, we opted for pgvector with 1536-dimensional embeddings. This allows us to find insights based on conceptual similarity, not just keywords.
• HNSW Index: For efficient nearest-neighbor search on our vector embeddings.
• RLS Policy: Row-Level Security to ensure insights are only accessible to authorized projects and users.
• tsvector Trigger: To automatically populate a tsvector column, enabling hybrid search.
• pairedInsightId: A clever addition to link related "pain" and "solution" insights, providing a richer context.

The ML Expert also proposed a hybrid search strategy: 70% vector similarity + 30% full-text search. This ensures both conceptual relevance and keyword accuracy.

3. The UX Developer's Touch: Seamless Integration

For the system to be truly useful, the user experience had to be intuitive. Our UX Developer designed the interaction points:

• SaveInsightsDialog: After a workflow review is approved, a dialog appears, allowing users to select key points, assign them to a project, and add tags before saving them as WorkflowInsight records.
• MemoryPicker: In the workflow creation page, a new component allows users to search, filter, and select relevant insights. Selected insights appear as "chips" with a collapsible preview.
• {{memory}} Template Variable: The ultimate integration point. Selected insights are automatically formatted and injected into the workflow prompt where {{memory}} is placed, providing immediate, relevant context to the agent.

Key Milestones & "Done" List

Beyond the design, we hit some critical development milestones:

• Refinement Mode for Agents: A major win! We committed a recreateFromKeyPoint refinement mode (28b6f05). Now, instead of regenerating an entire output from scratch, the LLM receives its previous output along with specific review feedback. This iterative process, using a REFINE_SEPARATOR to strip/replace blocks, significantly improves agent efficiency and output quality.
• Type Safety & UI Polish: A small but important fix involved a Badge variant type error in src/app/(dashboard)/dashboard/discussions/[id]/page.tsx. Changing "outline" to "accent" (cb04936) brought our type checks back to a fully clean state.

With the design complete, our ML Expert and UX Developer agents powered down, leaving the Architect to implement the backend plumbing for WorkflowInsight.

Navigating the Trenches: Unexpected Detours

Even with careful planning, development always throws a few curveballs. These "pain points" often become valuable lessons learned:

• Zsh Globbing with npx tsx: We ran into issues trying to execute npx tsx -e '...' with code containing ! (e.g., process.exit(!0)). Zsh's glob expansion interpreted the ! incorrectly, leading to esbuild syntax errors.
  • Workaround: Instead of in-line execution, we resorted to writing the script to a temporary file on disk and running it via npx tsx /path/to/temp/file.ts.
• Zsh Globbing with git add: Similarly, git add with paths containing brackets, like src/app/(dashboard)/dashboard/discussions/[id]/page.tsx, was being glob-expanded by Zsh.
  • Workaround: Simply quoting the path git add "src/app/(dashboard)/dashboard/discussions/[id]/page.tsx" resolved this.

These small shell quirks can eat up surprising amounts of time, reminding us to always be mindful of environment interactions!

What's Next: Bringing It All to Life

Currently, our development server is humming along, and we've confirmed the refinement mode is fully functional with a test workflow. The Architect is now deep into implementing Task #4:

1. Backend Implementation: Create the WorkflowInsight Prisma model, workflow-insights.ts service, saveInsights and insights.search tRPC procedures, and integrate the {{memory}} template variable into the workflow engine.
2. Database Updates: Run npm run db:push && npm run db:generate to apply schema changes and add RLS SQL for the workflow_insights table.
3. UI Development: Spawn a UI developer for Task #5 to build the SaveInsightsDialog, MemoryPicker, and MemoryContextPreview components, integrating them into the workflow detail and creation pages.
4. End-to-End Testing: Crucially, we'll test the full flow: run a workflow, review it, approve with saved insights, create a new workflow using the memory picker, and verify the {{memory}} injection.
5. Refinement Verification: User testing of the per-item recreate feature in the browser.

Looking further ahead into Phase 2, we'll be tackling the advanced aspects: setting up the pgvector extension, implementing an embedding service (supporting BYOK), optimizing the HNSW index, and fine-tuning the hybrid search algorithm.

This journey to build an intelligent workflow memory system is a testament to the power of structured problem-solving, collaborative agent design, and a willingness to tackle unexpected technical challenges head-on. We're excited to see how this system will empower our agents to become even smarter and more effective, transforming transient experiences into lasting, actionable knowledge.