← nyxcore-systems
8 min read

Unleashing AI Creativity: Building a Dynamic Workflow Engine from Scratch

We just replaced our rigid, hardcoded AI workflow with a powerful, dynamic engine. Dive into how we built a flexible, user-configurable system for chaining AI steps, complete with a drag-and-drop builder, live execution, and cost estimation.

workflow-engineaillmfull-stacknextjsprismatypescriptdevelopmentproduct-development

Every developer knows the pain of a hardcoded system. It starts simple, meets immediate needs, but quickly becomes a straitjacket as requirements evolve. We faced this exact challenge with our AI-powered application. Our initial system relied on a fixed, 5-step workflow – a rigid pipeline that, while functional, severely limited our ability to innovate and offer users true flexibility.

The goal was ambitious: to tear down the walls of that fixed workflow and build a full, dynamic workflow builder and execution engine. Imagine a world where users could design their own multi-step AI pipelines, chaining prompts, integrating different models, and even pausing for human review. That's the world we just brought to life.

After an intense development session, I'm thrilled to share that all 8 phases of this monumental project are complete. From schema evolution to a brand-new UI, we've laid the foundation for an infinitely more flexible and powerful AI experience.

The Vision: From Fixed Paths to Infinite Possibilities

Our old system dictated a specific sequence of AI interactions. If you wanted to deviate, you were out of luck. The new vision was clear: empower users to define any sequence of AI steps, configure each step down to the model and temperature, and observe the magic unfold in real-time. This meant building:

  1. A flexible data model to represent dynamic workflows and their individual steps.
  2. A robust execution engine capable of running these dynamic chains, handling pauses, retries, and variable resolution.
  3. A comprehensive API to manage workflows, steps, and templates.
  4. An intuitive UI for building, configuring, and monitoring these workflows.

Let's dive into how we made it happen.

The Journey: Building the Dynamic Workflow Engine

This wasn't a small undertaking. It involved touching nearly every part of our stack, from the database to the frontend. Here's a breakdown of the key phases:

1. Laying the Foundation: Schema Evolution

The first step was to re-imagine our database. Our WorkflowStep model needed to become infinitely more configurable. We added 13 new fields to WorkflowStep (e.g., label, prompt, systemPrompt, provider, model, temperature, maxTokens, stepType, enabled, retryCount, maxRetries, durationMs, tokenUsage, costEstimate, errorMessage).

Crucially, we introduced a WorkflowTemplate model, allowing us to define reusable blueprints. We also made userId optional and added a description to the main Workflow model, enabling both personal and shared workflows. Relations were added to link users and tenants to their respective workflows and templates. This schema overhaul was the bedrock for all subsequent flexibility.

prisma
// Example snippet from prisma/schema.prisma
model WorkflowStep {
  id           String    @id @default(uuid())
  workflowId   String
  workflow     Workflow  @relation(fields: [workflowId], references: [id], onDelete: Cascade)
  label        String    @default("") // New
  prompt       String    @default("") // New
  systemPrompt String?   // New
  provider     String?   // New
  model        String?   // New
  temperature  Float?    @default(0.7) // New
  maxTokens    Int?      @default(1024) // New
  stepType     String    @default("llm") // New
  enabled      Boolean   @default(true) // New
  // ... more fields for retries, timing, cost, etc.
}

model WorkflowTemplate {
  id          String   @id @default(uuid())
  tenantId    String
  name        String
  description String?
  isBuiltIn   Boolean  @default(false)
  config      Json     // Stores the template configuration
  tenant      Tenant   @relation(fields: [tenantId], references: [id])
}

2. Sharpening the Tools: Shared LLM Provider Resolution

One of our core features is "Bring Your Own Key" (BYOK) for LLM providers. To ensure consistency and avoid duplication, we extracted the resolveProvider logic into a shared service (src/server/services/llm/resolve-provider.ts). Now, any part of our backend can consistently resolve a user's chosen LLM provider, regardless of where the call originates. This modularity is key for maintainability.

3. Defining the Building Blocks: Constants Update

We formalized our WORKFLOW_STEP_TYPES and introduced a StepTemplate interface. This allowed us to define 16 distinct STEP_TEMPLATES (7 basic and 9 for our "Deep Build Pipeline" demo) and 3 BUILT_IN_WORKFLOW_TEMPLATES. These constants serve as the blueprint for both the backend engine and the frontend builder, ensuring a single source of truth for available step types and their default configurations.

4. The Brain of the Operation: Workflow Engine Rewrite

This was the heart of the project. The src/server/services/workflow-engine.ts underwent a complete rewrite (now 379 lines of pure logic!). Key features include:

  • ChainContext: A dynamic map to hold outputs and labels from previous steps, crucial for chaining.
  • resolvePrompt(): Our templating engine. This function intelligently resolves variables like {{input}}, {{input.field}}, and most importantly, {{steps.Label.content}} or {{steps.Label.field}}, allowing steps to feed directly into each other.
  • buildChainContext(): Reconstructs the execution context from the database, enabling seamless resume-after-pause.
  • executeStep(): Handles the execution of an individual step, including BYOK provider resolution, LLM calls, and error handling.
  • runWorkflow(): The orchestrator. Implemented as an AsyncGenerator, it handles:
    • Skipping already completed or disabled steps.
    • Pausing at "review" steps (our "non-YOLO" mode).
    • Retries with exponential backoff up to maxRetries.
    • Emitting real-time events (progress | text | done | error | retry | skipped | paused) with detailed metrics like tokenUsage, costEstimate, and durationMs.
  • estimateWorkflowCost(): Provides a pre-execution cost estimate using our COST_RATES, giving users transparency before they run expensive workflows.

5. The Nerve Center: tRPC Router Rewrite

Our src/server/trpc/routers/workflows.ts became significantly more robust (569 lines). It now features:

  • Dedicated sub-routers for steps (add, update, remove, reorder, retry, overrideOutput) and templates (list, create).
  • A comprehensive main router for list, get, create (supporting dynamic steps or template loading), update, start, resume, pause, duplicate, estimateCost, and delete workflows.
  • Crucially, we integrated LLM rate limiting into start, resume, and steps.retry to prevent abuse and manage resource usage.

6. The User's Canvas: Workflow Builder Page (NEW)

This is where the magic truly comes alive for the user. Our new src/app/(dashboard)/dashboard/workflows/new/page.tsx (686 lines) provides an intuitive interface:

  • Inputs for workflow name, description, and "YOLO" (run without pauses) mode.
  • A template picker with our built-in templates.
  • A sortable step list powered by @dnd-kit/sortable, allowing users to reorder steps with simple drag-and-drop.
  • Collapsible step cards, each offering granular control: step type, provider, model, temperature, max tokens, prompt, system prompt, enable toggle, and delete.
  • Quick-add buttons for our 7 basic step templates.
  • A sticky submit footer for mobile-friendliness.

This builder transforms a complex concept into an accessible, interactive experience.

7. The Live Show: Execution Page Rewrite

Monitoring a running AI workflow needs to be dynamic and informative. The src/app/(dashboard)/dashboard/workflows/[id]/page.tsx (540 lines) delivers:

  • A vertical pipeline visualization with status dots and connecting lines, showing the real-time progress.
  • Live Server-Sent Events (SSE) streaming, accumulating text output per step as it's generated.
  • MarkdownRenderer for beautifully formatted completed step output.
  • An inline edit output feature (textarea overlay) to allow users to modify a step's output and influence subsequent steps.
  • A retry button on failed or completed steps.
  • A settings panel for toggling YOLO mode, viewing cost estimates, cloning, or deleting the workflow.
  • Context-sensitive action buttons (Run/Pause/Resume) that adapt to the workflow's current state.
  • Leveraging Next.js 15's async params (React.use(params)) for cleaner data fetching.

8. The Overview: List Page Update

Finally, our src/app/(dashboard)/dashboard/workflows/page.tsx was updated to reflect the new capabilities. The "New" button now links directly to the builder (/workflows/new), and the list shows descriptions, step counts, and step labels in tooltips, providing more context at a glance.

Showcasing the Power: The Deep Build Pipeline Demo

To truly demonstrate the engine's capabilities, we built a sophisticated 9-step chained pipeline called "Deep Build Pipeline." This template takes an initial idea and iteratively refines it:

  1. Idea Generation
  2. Research & Brainstorm
  3. Add Features & Requirements
  4. Review 1 (Pause)
  5. Extend & Improve
  6. Review 2 (Pause)
  7. Project Wisdom (Reflect & Summarize)
  8. Improve & Refine
  9. Implementation Prompts

Each step has tailored system prompts and intelligently chains its output using {{steps.Label.content}}. The two "Review" gates automatically pause the workflow for human approval (unless YOLO mode is enabled), allowing users to intervene and guide the AI's progression. The final step generates self-contained AI coding prompts, ordered by dependency, ready for development. This template is a testament to the flexibility and power of the new engine.

Lessons Learned: Navigating the Development Minefield

No large-scale feature is built without its share of challenges. Here are a few notable "pains" that turned into valuable lessons:

  • Prisma Migrations with Required Fields: When adding required fields like label and prompt to WorkflowStep (which had existing rows), Prisma refused to db push without a default value.
    • Solution: We added @default("") to the new required fields and made userId optional (String? + User? relation) initially. Then, we ran a targeted backfill script using npx tsx -e to populate existing rows with meaningful data post-migration. Always consider existing data when adding new required fields!
  • Prisma JSON Null Handling: We encountered Type 'null' is not assignable to type 'NullableJsonNullValueInput | InputJsonValue' when trying to set a JSON field to null in a workflowStep.update operation.
    • Solution: Prisma has a specific Prisma.JsonNull constant for this purpose. Using Prisma.JsonNull correctly signals the intent to set a JSON field to SQL NULL.
  • TypeScript Map Iteration: When iterating over a Map using for (const [k, v] of map), TypeScript threw an error about needing --downlevelIteration.
    • Solution: The simplest workaround without changing tsconfig for this specific case was to convert the Map to an array first: Array.from(map.entries()).

What's Next?

With the core engine and UI complete, the immediate next steps involve rigorous testing of the full flow:

  • Creating and running the "Deep Build Pipeline" template.
  • Verifying output chaining and prompt variable resolution.
  • Testing the review step pause/resume cycle.
  • Confirming retry functionality on failed steps and its impact on subsequent steps.
  • Validating inline output editing and its influence on the chain.
  • Ensuring drag-and-drop reordering persists after saving.
  • Testing workflow duplication for deep copies of all step configurations.

This dynamic workflow engine is a game-changer for our platform, unlocking unprecedented flexibility and power for our users to build sophisticated AI applications. The future of AI interaction is here, and it's fully customizable.