← nyxcore-systems
7 min read

Taming Prisma's Type System: The Final Push for Our Dynamic Workflow Engine

We're on the cusp of launching our ambitious dynamic workflow engine, complete with 'Project Wisdom' integration. But before the finish line, a subtle yet critical Prisma bug threatened to derail our progress. Here's how we diagnosed, fixed, and learned from it.

PrismaTypeScriptWorkflowEngineBackendtRPCDatabaseBugFixDeveloperExperience

It's been a marathon, but we're finally staring down the barrel of full end-to-end testing for our new Dynamic Workflow Builder and Execution Engine. This isn't just any workflow tool; it's designed to be the brain of our operations, capable of adapting to complex processes and, crucially, injecting 'Project Wisdom' (our internal term for consolidated knowledge bases) directly into the execution flow.

All the features are in, type-checking is clean, and the builder UI feels solid. We were ready to hit the "go" button on our comprehensive test suite, but as always, the universe had one last little surprise waiting for us.

The Journey So Far: Building a Brain for Our Operations

Before diving into the last-minute heroics, let's quickly recap the monumental effort that got us here. This engine has been a multi-phase beast:

  • Schema Evolution (Phases 1-3): We laid the groundwork with 13 new WorkflowStep fields, introduced a WorkflowTemplate model, and added crucial userId, description, and consolidationIds fields to our Workflow schema. We also built a shared llm/resolve-provider.ts for AI integration and updated our core constants.
  • Engine Rewrite (Phases 4-5): This was the heart transplant. We rewrote the core engine to support dynamic templating ({{input}}, {{steps.Label.content}}, {{consolidations}}), robust retry mechanisms, resume capabilities, and precise cost estimation. Our tRPC router grew to support all these new functionalities, including duplicate, resume, estimateCost, and delete mutations.
  • Builder & Execution UI (Phases 6-8): The front-end came alive! A brand-new /workflows/new page featuring dnd-kit for intuitive drag-and-drop workflow building, complete with a consolidation picker. The execution page got a live pipeline visualization with SSE streaming, MarkdownRenderer for rich output, and controls for editing, retrying, and managing linked consolidations. Even the list page got a facelift.
  • Project Wisdom Integration: This is where the magic happens. By adding consolidationIds to the Workflow schema, our engine can now call loadConsolidationContent() (which in turn uses generatePromptHints()) to dynamically inject relevant "Project Wisdom" into steps, especially our Deep Build Pipeline.

Everything was wired up. The dream of dynamic, knowledge-infused workflows was within reach.

The Final Hurdle: Taming Prisma's Type System

Just as we were gearing up for the final end-to-end tests, a persistent and perplexing error reared its head during workflow creation:

Invalid prisma.workflow.create() error — `userId` was passed as a scalar field alongside nested `steps: { create: [...] }`, which forced Prisma to resolve the "checked" input type where `userId` is not a valid key.

The specific error message from Prisma was something along the lines of:

Unknown argument 'userId'. Available options are marked with ?.

This was baffling. userId is clearly a foreign key on our Workflow model, and we've been using it successfully elsewhere. What gives?

The "Why": Prisma's Input Type XORing

After some head-scratching and diving deep into Prisma's documentation (and a bit of trial-and-error), the culprit became clear: Prisma's clever but sometimes tricky input type resolution.

When you perform a create operation, Prisma can infer two main types of input:

  1. WorkflowCreateInput (Checked Input): This expects relation fields for foreign keys. So, instead of userId: 'some-id', it expects user: { connect: { id: 'some-id' } }.
  2. WorkflowUncheckedCreateInput (Unchecked Input): This expects scalar foreign keys. Here, userId: 'some-id' is perfectly valid.

The catch? When you include nested relation writes (like our steps: { create: [...] }) within the same create operation, Prisma forces the input to be of the "checked" type (WorkflowCreateInput). This means that suddenly, directly passing userId as a scalar is no longer valid, because in the checked type, userId is not a direct field; user is the relation field.

Our problematic code in src/server/trpc/routers/workflows.ts looked something like this (simplified):

typescript
// Problematic prisma.workflow.create() call
prisma.workflow.create({
  data: {
    name: input.name,
    description: input.description,
    userId: ctx.user.id,     // Scalar foreign key
    tenantId: ctx.tenantId,   // Scalar foreign key
    steps: {
      create: input.steps.map(step => ({ /* nested step creation data */ }))
    },
    // ... other fields
  },
});

The Fix: Embracing Relational Connects

The solution, once understood, was straightforward: switch to the relational connect syntax for all foreign keys when performing nested relation writes.

typescript
// The fix: Using relational connect syntax
prisma.workflow.create({
  data: {
    name: input.name,
    description: input.description,
    user: { connect: { id: ctx.user.id } },     // Relational connect
    tenant: { connect: { id: ctx.tenantId } },   // Relational connect
    steps: {
      create: input.steps.map(step => ({ /* nested step creation data */ }))
    },
    // ... other fields
  },
});

We applied the same fix to our duplicate mutation, which also involved nested step creation. With this change, the errors vanished, and our type-checking remained green.

The Golden Rule: When using nested relation writes (like create, connect, set) within a Prisma create or update operation, always use the relational { connect: { id } } or { create: {} } syntax for all your foreign key relationships. Mixing scalar foreign keys (userId: someId) with nested relation writes will lead to Prisma's type system getting confused and throwing these Unknown argument errors.

More Wisdom from the Trenches

This wasn't our first rodeo with tricky edge cases. A few other lessons learned from earlier sessions that are worth reiterating:

  • Prisma.JsonNull vs. null: When resetting Json fields in Prisma, remember to use Prisma.JsonNull instead of plain null. null will often be interpreted as NULL in the database, whereas JsonNull correctly represents a JSON null value.
  • Map Iteration in Older TS: If you're targeting an older TypeScript version or compilation environment, iterating directly over Map objects with for...of might trigger --downlevelIteration errors. A robust workaround is Array.from(map.entries()).
  • New Required Columns: When adding new required columns to an existing database model, always make them optional initially or add an @default("") to avoid breaking existing rows during migrations.

What's Next: The Finish Line in Sight

With the Prisma bug squashed and our client regenerated, the path is clear for final validation. Our immediate next steps are:

  1. Restart the dev server: npm run dev to pick up the Prisma client changes.
  2. Create a workflow: We'll use the Deep Build Pipeline template and link a consolidation to verify the fix.
  3. Run the workflow: Crucially, we need to confirm that {{consolidations}} resolves correctly in the Project Wisdom step prompt.
  4. Full E2E Test: We'll put the entire 9-step pipeline through its paces: SSE streaming, review pauses, retry functionality, editing output, and checking the settings panel.
  5. No Consolidation Test: Verify the fallback text appears correctly when no consolidations are linked.
  6. Enhance UI (Future): Consider enriching the execution page's consolidation display to show names (currently it just shows generic badges as only IDs are stored on the workflow).

It's exciting to be at this stage. The Dynamic Workflow Builder + Execution Engine, powered by Project Wisdom, is poised to be a game-changer for how we manage and execute complex tasks. On to full end-to-end testing!

json
{
  "thingsDone": [
    "Implemented Dynamic Workflow Builder and Execution Engine",
    "Integrated Project Wisdom (consolidation) into workflow execution",
    "Fixed critical Prisma runtime `Invalid prisma.workflow.create()` bug",
    "Completed schema evolution for Workflow and WorkflowStep models",
    "Rewrote workflow engine with templating, retry, resume, and cost estimation",
    "Developed builder UI with dnd-kit and execution UI with SSE streaming",
    "Applied Prisma fix to both create and duplicate mutations"
  ],
  "pains": [
    "Prisma's `Unknown argument 'userId'` error due to mixing scalar FKs with nested relation creates",
    "Prisma's input type XORing (Checked vs. Unchecked) confusion",
    "Remembering to use `Prisma.JsonNull` instead of `null` for JSON fields",
    "Handling `for...of` iteration on Maps for older TypeScript targets (`--downlevelIteration`)",
    "Managing new required database columns in existing tables during migrations"
  ],
  "successes": [
    "Successfully diagnosed and fixed the subtle Prisma type-checking issue",
    "Enabled full end-to-end testing for the entire workflow engine",
    "Completed core feature set for the dynamic workflow builder and execution engine",
    "Gained deeper understanding of Prisma's input type resolution mechanisms"
  ],
  "techStack": [
    "Prisma (v5.22.0)",
    "TypeScript",
    "tRPC",
    "PostgreSQL",
    "Next.js (implied by /app/(dashboard) structure)",
    "dnd-kit",
    "Server-Sent Events (SSE)",
    "MarkdownRenderer"
  ]
}