← nyxcore-systems
8 min read

Beyond the Prompt: Building a Smarter, Safer, and Testable AI Workflow Engine

We just shipped a major update to our AI workflow engine, focusing on deep context injection, robust injection diagnostics, persona A/B testing, and critical security hardening. Here's a deep dive into the how and why.

AILLMWorkflowEngineContextManagementPromptEngineeringSecurityA/BTestingTypeScriptPrismatRPCNext.js

Just wrapped up a late-night coding session on 2026-02-25, pushing a chunky commit (5ab0a15 to main) that significantly upgrades our AI workflow engine. The goal for this session was ambitious: to make our workflows more intelligent, transparent, and resilient. We tackled four key areas: injecting context-aware data, providing diagnostics for prompt injection, enabling persona A/B testing, and shoring up security.

Building an AI-powered system isn't just about calling an LLM API. It's about orchestrating data, managing state, understanding user intent, and ensuring reliability. This session was all about refining that orchestration.

1. The Quest for Context-Aware Workflows: Enriching the Pipeline

Our AI workflow engine thrives on context. Generic prompts yield generic results. To make our workflows truly valuable, they need to be deeply aware of the project, user, and historical data they're operating within.

The Problem: Manually linking every piece of relevant information to a workflow step is tedious and error-prone. We needed a way for the system to intelligently discover and inject context.

Our Solution: Automatic Pipeline Enrichment. We modified the createWorkflow mutation in src/server/trpc/routers/action-points.ts. Now, when a new workflow is initiated, it automatically discovers and fetches related consolidations, insights, personas, and repository information for the linked project. This is done efficiently using Promise.all to fetch data concurrently, ensuring the workflow starts with a rich, relevant context.

typescript
// src/server/trpc/routers/action-points.ts (simplified)
createWorkflow: protectedProcedure
  .input(createWorkflowSchema)
  .mutation(async ({ ctx, input }) => {
    // ... initial workflow creation logic ...

    // Auto-discover context for the linked project
    const [consolidations, insights, personas, repos] = await Promise.all([
      ctx.db.consolidation.findMany({ where: { projectId: input.projectId } }),
      ctx.db.insight.findMany({ where: { projectId: input.projectId } }),
      // ... more context fetching ...
    ]);

    // ... then, use this context to build initial step prompts ...
  });

Crucially, we also enriched our step prompts with new template variables: {{project.wisdom}} and {{memory}}. This allows workflow authors to directly reference high-level project knowledge and specific memory snippets, making prompts far more dynamic and effective.

Lesson Learned: Adapting to Schema Realities

Initially, the plan was to filter personas by a tags field. However, a quick check of prisma/schema.prisma revealed the Persona model didn't have a tags field. This is a common hiccup in agile development – plans meet reality.

Workaround: Instead of halting, we adapted. Persona matching now uses text matching against the persona's name and description fields against the action point category. This works perfectly for category-based persona selection, and we've noted that if more granular tagging is needed in the future, adding tags String[] to the Persona model is a straightforward update. This flexibility is key when iterating quickly.

2. Shining a Light on LLM Inputs: Injection Diagnostics

"Garbage in, garbage out" is especially true for LLMs. When constructing complex prompts with multiple context sources, it's incredibly easy for variables to be unresolved, content to exceed token limits, or for unexpected formatting issues to creep in. Debugging these "invisible" inputs is a major pain point.

The Problem: Lack of visibility into what context actually gets injected into an LLM prompt. Are all the {{variables}} resolving? How much content is being sent?

Our Solution: A Dedicated Injection Diagnostics Service. We introduced src/server/services/injection-diagnostics.ts, a new service designed to inspect and report on the context provided to an LLM.

Key features include:

  • ContextSource / InjectionReport interfaces: Standardized structures to describe context and its diagnostic findings.
  • measureContextSources(ctx): Inspects various ChainContext fields, providing character and estimated token counts for each source. This is vital for managing token budget.
  • detectUnresolvedVariables(prompt): A regex scan for our placeholder patterns like [No ... linked], immediately highlighting any context that failed to resolve.
  • buildInjectionReport(...): Assembles a comprehensive report for a given step.
  • sanitizeContextContent(content): (More on this in security, but it's also used here!) Escapes {{ to \{\{ to prevent accidental template variable interpretation, and logs suspicious prompt override patterns.

This diagnostic report is integrated directly into our workflow-engine.ts. Every executeStep() now generates an injectionReport which is stored in the step's checkpoint JSON and emitted as a context_report SSE event.

On the UI side (src/app/(dashboard)/dashboard/workflows/[id]/page.tsx), we've added a collapsible "Context Diagnostics" panel to completed steps. This panel displays a table showing each context source, its estimated token count, and a status indicator (green/yellow/red dots). Unresolved variables are prominently displayed in orange. This gives developers and users an unprecedented level of transparency into how their prompts are being constructed and executed.

3. A/B Testing for AI Personas: Finding the Right Voice

Different LLM personas can dramatically alter the tone, style, and effectiveness of an AI's output. Comparing these variations systematically is crucial for optimizing workflows.

The Problem: How do we easily compare the output of multiple personas for a single workflow step without creating multiple, separate workflows?

Our Solution: First-Class Persona A/B Testing. We extended our WorkflowStep model in prisma/schema.prisma to include a new field: comparePersonas String[] @default([]) @db.Uuid.

prisma
// prisma/schema.prisma
model WorkflowStep {
  // ... other fields ...
  comparePersonas String[] @default([]) @db.Uuid
}

After running npm run db:push to sync the schema, we updated our tRPC routers (src/server/trpc/routers/workflows.ts) to allow comparePersonas in step updates and duplications.

The core logic lives in workflow-engine.ts. We expanded the condition for generating alternatives: step.generateCount > 1 || step.comparePersonas.length > 1. Now, if comparePersonas is specified, the engine will:

  1. Prioritize persona comparisons over other variation strategies (like provider comparisons).
  2. Load each specified comparison persona.
  3. Execute executeStep once for each persona, overriding the default personaId.
  4. Crucially, it also runs a "No Persona (Baseline)" comparison by temporarily clearing the personaSystemPrompts, giving us a neutral benchmark.

In the UI, users can now multi-select personas in the step configuration area. The system then displays the count of selected personas plus a "baseline" run, and presents the results in distinct alternative tabs, making side-by-side comparison intuitive. This empowers users to iterate on persona design and find the most effective "voice" for their AI.

4. Fortifying the Gates: Security Hardening for Context Injection

As we inject more dynamic content into LLM prompts, the surface area for potential prompt injection attacks increases. We need to be vigilant.

The Problem: Untrusted or user-generated content, when injected into prompts, could lead to prompt injection attacks or unintended LLM behavior.

Our Solution: Context Sanitization at the Source. We leveraged and extended the sanitizeContextContent() function from our injection-diagnostics.ts service for this purpose.

This function now handles:

  • Template variable injection prevention: It escapes literal {{ sequences found within context content to \{\{. This prevents data from being accidentally (or maliciously) interpreted as a new template variable within the prompt.
  • Suspicious pattern detection: It logs warnings for patterns commonly associated with prompt injection or attempts to override system instructions (e.g., "ignore previous instructions", "[SYSTEM]", "as an AI model, you must..."). While not a full prevention, it provides an audit trail and alerts for suspicious activity.
typescript
// src/server/services/injection-diagnostics.ts (simplified)
export function sanitizeContextContent(content: string): string {
  // Escape potential template variables to prevent injection
  let sanitized = content.replace(/\{\{/g, '\\{\\{');

  // Log suspicious patterns that might indicate prompt injection attempts
  const suspiciousPatterns = [
    /ignore previous instructions/i,
    /\[SYSTEM\]/i,
    /you must/i,
    // ... more patterns
  ];
  for (const pattern of suspiciousPatterns) {
    if (pattern.test(content)) {
      console.warn(`Suspicious pattern detected in context: "${content.substring(0, 100)}..."`);
      // Potentially redact or further sanitize based on severity
    }
  }
  return sanitized;
}

This sanitization is integrated into resolvePrompt() and applied specifically to template variables that pull potentially untrusted or dynamic content: consolidations, memory, project.wisdom, claudemd, and docs.

Important Note: We deliberately do not apply this sanitization to fileTree (as it's structural and trusted) or to the step.prompt itself (as that's directly authored by the user and expected to contain template variables). This nuanced approach ensures we protect against injection where it matters most, without breaking intended functionality.

Immediate Next Steps & Remaining Tasks

While the main features are in, a developer's work is never truly done! Here's what's next on the plate:

  1. GitHub Org Repo Fetching: A user asked about fetching repos from the clarait GitHub org. We need to modify fetchRepos() in src/server/services/github-connector.ts to either add affiliation=owner,collaborator,organization_member or use the dedicated /orgs/{org}/repos API call.
  2. Persona CRUD Enhancements: Commit the remaining persona CRUD changes (personas/new/page.tsx, persona-generator.ts, personas.ts) from prior sessions.
  3. Cleanup: Remove untracked .log files from the project root and add them to .gitignore.
  4. Full Pipeline Testing: Execute a full action-point-to-workflow pipeline: create an action point on a project with consolidations, verify the workflow gets non-empty context IDs, run it, and check the output references project patterns.
  5. Persona A/B Testing Validation: Set comparePersonas on a step with 2+ personas, run it, and verify that the alternatives tabs correctly show persona names and outputs.
  6. Security Validation: Create a memory insight containing {{consolidations}}, run a workflow that uses it, and verify that {{ was correctly escaped in the prompt output.

Wrapping Up

This session was a significant leap forward for our AI workflow engine. By focusing on deep context injection, providing transparent diagnostics, enabling systematic persona testing, and hardening security, we're not just building features – we're building a more intelligent, reliable, and user-friendly platform for AI-powered operations. The journey continues, but the foundations are getting stronger with every commit.

What are your strategies for managing context, debugging LLM inputs, or A/B testing personas in your AI applications? I'd love to hear them!

json
{
  "thingsDone": [
    "Implemented context-aware action-point workflows (pipeline enrichment)",
    "Developed robust injection diagnostics for LLM prompts",
    "Enabled persona A/B testing within workflow steps",
    "Applied security hardening for context injection to prevent prompt injection"
  ],
  "pains": [
    "Initial plan to filter personas by a 'tags' field failed because the Prisma model lacked that field, requiring a workaround using name/description text matching."
  ],
  "successes": [
    "All four planned features were implemented, type-check clean, committed, and pushed.",
    "Successfully adapted persona matching strategy when schema limitations were encountered.",
    "Developed a comprehensive diagnostic service providing deep visibility into LLM context.",
    "Integrated A/B testing seamlessly into the workflow engine and UI.",
    "Implemented proactive security measures for context sanitization."
  ],
  "techStack": [
    "TypeScript",
    "Next.js",
    "tRPC",
    "Prisma",
    "PostgreSQL",
    "LLMs (Language Models)",
    "SSE (Server-Sent Events)"
  ]
}