Building a Dynamic Workflow Engine: From Hardcoded Steps to Full Flexibility

Ever found yourself stuck with a workflow system that's too rigid? That's exactly where we were with our 5-step hardcoded workflow system. What started as a simple automation tool quickly hit the walls of inflexibility when users wanted custom steps, different AI models, and complex chaining logic.

So we decided to rebuild it from the ground up. Here's the story of how we transformed a static workflow system into a fully dynamic workflow builder and execution engine.

The Vision: From Static to Dynamic

Our original system was straightforward but limiting:

5 hardcoded steps
No customization
Linear execution only
No visual builder

The new system needed to support:

Dynamic step creation with custom prompts and configurations
Visual workflow builder with drag-and-drop reordering
Template system for reusable workflow patterns
Real-time execution with streaming progress updates
Step chaining where outputs become inputs for subsequent steps
Pause/resume functionality for human review gates

Phase 1: Database Schema Evolution

First, we needed to evolve our database schema to support dynamic workflows. The original WorkflowStep model was bare-bones, so we expanded it significantly:

sql

model WorkflowStep {
  // Original fields
  id         String @id @default(cuid())
  workflowId String
  order      Int
  
  // New dynamic fields
  label         String @default("")
  prompt        String @default("")
  systemPrompt  String?
  provider      String @default("openai")
  model         String @default("gpt-4")
  temperature   Float @default(0.7)
  maxTokens     Int @default(2000)
  stepType      String @default("llm")
  enabled       Boolean @default(true)
  
  // Execution tracking
  retryCount    Int @default(0)
  maxRetries    Int @default(3)
  durationMs    Int?
  tokenUsage    Int?
  costEstimate  Float?
  errorMessage  String?
}

We also introduced a WorkflowTemplate system for reusable patterns:

sql

model WorkflowTemplate {
  id          String  @id @default(cuid())
  tenantId    String
  name        String
  description String?
  isBuiltIn   Boolean @default(false)
  config      Json    // Stores step configurations
}

Phase 2: The Heart - Workflow Engine Rewrite

The core execution engine needed a complete rewrite to handle dynamic workflows. Here's what we built:

Template Variable Resolution

One of the coolest features is our template variable system. Steps can reference:

{{input}} - The original workflow input
{{input.field}} - Specific fields from structured input
{{steps.Label.content}} - Output from previous steps
{{steps.Label.field}} - Specific fields from previous step outputs

typescript

function resolvePrompt(template: string, context: ChainContext): string {
  return template.replace(/\{\{([^}]+)\}\}/g, (match, path) => {
    const parts = path.trim().split('.');
    
    if (parts[0] === 'input') {
      return parts.length === 1 ? context.input : 
             getNestedValue(context.input, parts.slice(1));
    }
    
    if (parts[0] === 'steps' && parts.length >= 2) {
      const stepLabel = parts[1];
      const stepOutput = context.stepOutputs.get(stepLabel);
      return parts.length === 2 ? stepOutput?.content || '' :
             getNestedValue(stepOutput, parts.slice(2));
    }
    
    return match;
  });
}

Streaming Execution with Events

The engine runs as an async generator, yielding events for real-time UI updates:

typescript

async function* runWorkflow(workflowId: string, input: any) {
  const workflow = await getWorkflowWithSteps(workflowId);
  const context = await buildChainContext(workflow, input);
  
  for (const step of workflow.steps) {
    if (!step.enabled) {
      yield { type: 'skipped', stepId: step.id, stepLabel: step.label };
      continue;
    }
    
    // Pause at review steps (unless YOLO mode)
    if (step.stepType === 'review' && !workflow.yoloMode) {
      yield { type: 'paused', stepId: step.id, stepLabel: step.label };
      return;
    }
    
    yield { type: 'progress', stepId: step.id, stepLabel: step.label };
    
    try {
      const result = await executeStep(step, context);
      context.stepOutputs.set(step.label, result);
      
      yield { 
        type: 'text', 
        stepId: step.id, 
        stepLabel: step.label,
        content: result.content,
        tokenUsage: result.tokenUsage,
        costEstimate: result.costEstimate
      };
    } catch (error) {
      if (step.retryCount < step.maxRetries) {
        yield { type: 'retry', stepId: step.id, attempt: step.retryCount + 1 };
        // Exponential backoff retry logic...
      } else {
        yield { type: 'error', stepId: step.id, error: error.message };
      }
    }
  }
  
  yield { type: 'done' };
}

Phase 3: Visual Workflow Builder

Building the UI was where things got really interesting. We created a drag-and-drop workflow builder using @dnd-kit/sortable:

tsx

function WorkflowBuilder() {
  const [steps, setSteps] = useState<WorkflowStep[]>([]);
  
  return (
    <DndContext onDragEnd={handleDragEnd}>
      <SortableContext items={steps.map(s => s.id)}>
        {steps.map((step, index) => (
          <SortableStepCard 
            key={step.id} 
            step={step} 
            index={index}
            onUpdate={updateStep}
            onDelete={deleteStep}
          />
        ))}
      </SortableContext>
    </DndContext>
  );
}

Each step card is collapsible and includes:

Step type selector (LLM, Review, Transform, etc.)
Provider and model selection
Temperature and token limit controls
Prompt and system prompt editors
Enable/disable toggle

Phase 4: Real-time Execution Interface

The execution page shows a vertical pipeline with live updates via Server-Sent Events:

tsx

function WorkflowExecution({ workflowId }: { workflowId: string }) {
  const [stepStates, setStepStates] = useState<Map<string, StepState>>(new Map());
  
  useEffect(() => {
    const eventSource = new EventSource(`/api/v1/events/workflows/${workflowId}`);
    
    eventSource.onmessage = (event) => {
      const data = JSON.parse(event.data);
      
      setStepStates(prev => new Map(prev).set(data.stepId, {
        status: data.type,
        content: data.content || '',
        tokenUsage: data.tokenUsage,
        costEstimate: data.costEstimate
      }));
    };
    
    return () => eventSource.close();
  }, [workflowId]);
  
  return (
    <div className="workflow-pipeline">
      {workflow.steps.map((step, index) => (
        <StepCard 
          key={step.id}
          step={step}
          state={stepStates.get(step.id)}
          showConnector={index < workflow.steps.length - 1}
        />
      ))}
    </div>
  );
}

Demo Template: Deep Build Pipeline

To showcase the system's power, we created a 9-step "Deep Build Pipeline" template:

Idea Analysis - Break down the core concept
Research - Gather relevant information
Feature Addition - Expand on the idea
Review Gate - Human approval checkpoint
Extension & Improvement - Refine the concept
Second Review Gate - Final human checkpoint
Project Wisdom - Add strategic insights
Final Improvements - Polish the output
Implementation Prompts - Generate actionable coding tasks

Each step chains its output to the next using our template variable system, creating a sophisticated pipeline that transforms a simple idea into detailed implementation guidance.

Lessons Learned: The Challenges We Faced

Database Migration Pain Points

Challenge: Adding required fields to existing tables with data.

sql

-- This failed: "Added required column without default value"
ALTER TABLE WorkflowStep ADD COLUMN label String NOT NULL;

Solution: Use temporary defaults, then backfill:

sql

ALTER TABLE WorkflowStep ADD COLUMN label String DEFAULT "";
-- Then run backfill script to populate real values

Prisma JSON Field Quirks

Challenge: TypeScript errors when setting JSON fields to null.

typescript

// This failed
workflowStep.update({ data: { config: null } });

Solution: Use Prisma's special null value:

typescript

workflowStep.update({ data: { config: Prisma.JsonNull } });

Map Iteration in TypeScript

Challenge: Direct Map iteration causing downlevelIteration errors.

typescript

// This required special tsconfig settings
for (const [key, value] of map) { ... }

Solution: Convert to array first:

typescript

for (const [key, value] of Array.from(map.entries())) { ... }

The Results

The transformation was dramatic:

Flexibility: Users can now create workflows with 1-50+ steps
Reusability: Template system enables sharing of proven patterns
Visibility: Real-time streaming shows exactly what's happening
Control: Pause/resume and retry capabilities for complex workflows
Cost Management: Pre-execution cost estimation and per-step tracking

What started as a rigid 5-step system became a powerful, flexible workflow engine that users actually want to use.

Next Steps

The foundation is solid, but we're already planning enhancements:

Conditional branching for dynamic workflow paths
Parallel execution for independent steps
External integrations beyond LLM providers
Workflow marketplace for community templates
Advanced debugging with step-by-step inspection

Building developer tools is all about removing friction while adding power. Sometimes that means throwing out what works and rebuilding from first principles. In this case, it was absolutely worth it.

Want to see the code in action? The complete implementation includes 8 phases of development, from database schema to UI components. The modular architecture makes it easy to extend with new step types and execution patterns.