Building an End-to-End Memory System: From Component Design to Template Injection

Ever wondered what it takes to build a system that can remember and inject contextual insights across complex workflows? Last week, I completed an end-to-end memory system that does exactly that - and the journey was full of interesting technical challenges and "aha!" moments.

The Vision: Contextual Memory for Workflows

The goal was ambitious but clear: create a system where users could save insights during workflow execution, search and select relevant memories when creating new workflows, and automatically inject that context using template variables like {{memory}}.

Think of it as giving your automation workflows a long-term memory that gets smarter over time.

The Architecture: Five Moving Parts

The complete system ended up having five main components that had to work together seamlessly:

1. MemoryPicker Component

// Core functionality: search, filter, and select insights
const MemoryPicker = () => {
  const { data: insights } = trpc.memory.listInsights.useQuery({
    search: searchTerm,
    categories: selectedCategories,
    severities: selectedSeverities
  });
  
  return (
    <div className="space-y-4">
      {/* Search and filters */}
      {insights?.map(insight => (
        <InsightCard 
          key={insight.id}
          insight={insight}
          onSelect={handleSelect}
        />
      ))}
    </div>
  );
};

This component handles the user experience of browsing and selecting memories. It includes search functionality, category filters, severity badges, and expandable detail views with {{memory}} preview.

2. SaveInsightsDialog Integration

The dialog that appears after workflow steps to capture key insights. This connects to the tRPC endpoint memory.saveInsights and includes proper step labeling and project association.

3. Template Injection System

The magic happens when {{memory}} in a workflow step gets replaced with actual insight content:

// Template processing that injects selected memories
const processedContent = stepContent.replace(
  /\{\{memory\}\}/g, 
  selectedInsights.map(insight => 
    `[${insight.severity}] ${insight.content}`
  ).join('\n\n')
);

4. Collapsible Context Sections

A UI improvement that organizes the workflow creation interface into collapsible sections (Consolidations, Personas, Docs, Memory) with badge counts showing selections.

5. Memory-Pull Script

A bash script for syncing memory files between environments:

#!/bin/bash
# scripts/memory-pull.sh - Fetch memory files without full merge
git fetch origin
git checkout origin/main -- .memory/

Lessons Learned: The Debugging Chronicles

The Case of the Missing Action Field

The Problem: After implementing everything, the SaveInsightsDialog wouldn't appear when clicking "Approve & Continue" after a review step.

The Investigation: The dialog was supposed to show when there were key points to save, but the filter condition kp.action === "keep" was failing.

The Revelation: The extractKeyPoints() function returns objects without an action field by default. In JavaScript, undefined !== "keep" evaluates to true, so our filter was rejecting all key points.

The Fix:

// Before (broken)
keyPoints.filter(kp => kp.action === "keep")

// After (working)
keyPoints.filter(kp => !kp.action || kp.action === "keep" || kp.action === "edit")

This was a classic case of defensive programming - handling the "default state" that we hadn't explicitly considered.

The Duplicate Save Problem

The Problem: Users clicking the save button multiple times resulted in duplicate database records (30 records instead of 10).

The Immediate Fix: Manual SQL cleanup to remove duplicates.

The Long-term Solution: Need to implement either button disabling after first click or mutation-level deduplication.

The Reserved Variable Trap

The Problem: A bash script using status as a variable name failed with "read-only variable: status".

The Learning: Different shells (bash vs zsh) have different reserved variables. status is reserved in zsh.

The Solution: Use more specific variable names like step_status to avoid conflicts.

The Moment of Truth: End-to-End Verification

The real test came when I created a "Memory Injection Test" workflow:

1. Selected 5 insights with different severity levels
2. Created a single step using the {{memory}} template variable
3. Executed the workflow

Result: The LLM received the injected insights and responded with an accurate, severity-tagged summary. The entire execution took 3.6 seconds and cost $0.0035.

Seeing those insights seamlessly flow from the database through template processing into the LLM context was incredibly satisfying.

What's Next: Phase 2 Improvements

With the core system working, the next phase focuses on:

1. Vector Similarity Search: Upgrading to pgvector for semantic insight matching
2. Project-Scoped Filtering: Ensuring insights are contextually relevant to specific projects
3. Built-in Template Integration: Adding {{memory}} to default step templates
4. Performance Optimization: Handling larger memory datasets efficiently

Key Takeaways

Building this memory system taught me several valuable lessons:

1. Default States Matter: Always consider what happens when optional fields are undefined
2. End-to-End Testing is Crucial: Individual components can work perfectly but fail when integrated
3. User Experience Details: Small UI improvements like collapsible sections and loading states make a big difference
4. Defensive Programming Pays Off: Handling edge cases upfront prevents debugging sessions later

The combination of React components, tRPC for type-safe APIs, and careful state management created a system that feels both powerful and intuitive to use. Most importantly, it actually works in production - which is always the ultimate test.

Have you built similar memory or context systems? I'd love to hear about your approach and the challenges you encountered. Feel free to reach out with questions about any of the technical details covered here.