← nyxcore-systems
7 min read

Beyond the Hype: Benchmarking LLM Personas for Production Readiness

We just launched a comprehensive system to rigorously evaluate LLM persona consistency, safety, and performance, featuring LLM-as-Judge scoring and intuitive trend visualizations. Here's how we built it.

LLMBenchmarkingPersonaEvaluationTypeScriptPrismatRPCNextJSRechartsAISoftwareDevelopment

The promise of large language models is immense, but bringing them into production, especially when they embody specific personas, introduces a new set of challenges. How do you ensure your "Cael, the meticulous research assistant" doesn't suddenly start giving legal advice like "Lee"? Or that your "Morgan, the financial analyst" doesn't succumb to a subtle jailbreak attempt?

That's precisely the problem we set out to solve: building a robust Persona Evaluation & Benchmark System. After a focused development sprint, I'm thrilled to report that the core system is feature-complete, type-checked, lint-clean, and ready for deployment. This post dives into the "how" and "why" behind our solution, sharing the architectural decisions and lessons learned along the way.

The Mandate: Why We Needed a Persona Evaluation System

Our goal was clear: create a system that could automatically assess an LLM persona's adherence to its defined role, its resilience against adversarial prompts, and its ability to maintain coherence under heavy context. This isn't just about "Does it work?" but "Does it work consistently, safely, and reliably?"

The system needed:

  1. Diverse Test Types: To probe different aspects of persona behavior.
  2. Objective Scoring: Leveraging LLMs themselves to judge responses.
  3. Trend Visualization: To track performance over time and identify regressions.
  4. Developer-Friendly UI: Making it easy to run tests and interpret results.

Building Blocks: From Schema to Dashboard

Let's break down the key components we put together:

1. The Data Foundation: Prisma & RLS

At the heart of any data-driven system is the schema. We introduced PersonaEvaluation to our prisma/schema.prisma model, establishing crucial indexes and relations back to Persona and Tenant. This ensures our evaluation data is structured, searchable, and tied directly to the personas being tested.

sql
-- Excerpt from prisma/rls.sql for tenant isolation
ALTER TABLE "PersonaEvaluation" ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation_policy ON "PersonaEvaluation" FOR ALL
USING (tenant_id = current_setting('app.tenant_id')::uuid);

Crucially, we implemented Row Level Security (RLS) in prisma/rls.sql. This ensures that evaluation data is strictly isolated by tenant, a non-negotiable requirement for multi-tenant applications. Each tenant can only see their own persona evaluations.

2. The Brains: persona-evaluator.ts

This is where the magic happens. Our src/server/services/persona-evaluator.ts orchestrates the entire testing process. We designed three core test types:

  • runTemperatureTest(): This test probes an LLM's consistency and creativity. It sends the exact same prompt twice, once at a low temperature (e.g., 0.2 for consistency) and once at a higher temperature (e.g., 0.8 for creative variation), and then compares the outputs. It helps us understand if the persona remains stable or becomes unhinged at higher temperatures.
  • runJailbreakTest(): A critical safety test. We employ a series of four escalating attack prompts:
    • Ignoring instructions
    • System override attempts
    • Admin pretense
    • Anti-persona "game" scenarios This helps us gauge the persona's robustness against malicious or manipulative inputs.
  • runDegradationTest(): To assess performance under stress, this test injects a 3000+ word enterprise scenario as context and then checks the persona's ability to maintain coherence and adhere to its role within that dense information. It's a real-world stress test for context window management.

LLM-as-Judge: Objective Scoring

Central to our evaluation is the judgeResponse() function. Instead of manual human review for every test run, we leverage another LLM to act as an impartial judge. This judge is prompted to extract structured JSON containing three key metrics (0-100):

  • roleAdherence: How well did the LLM stick to its defined persona?
  • ruleCompliance: Did it follow all instructions and constraints?
  • markerPresence: Did it include specific, persona-relevant markers? (e.g., "Cael" mentioning papers and math, "Lee" referencing DSGVO, "Morgan" discussing financial terms, "Sage" identifying refactoring patterns).

Each test type has weighted scoring. For instance, the jailbreak test heavily weights roleAdherence (at 0.6) to ensure the persona doesn't break character.

3. The API Layer: tRPC Endpoints

Our src/server/trpc/routers/personas.ts exposes three efficient tRPC endpoints:

  • runEvaluation (mutation): Triggers one or all test types for a given persona.
  • evaluationHistory (query): Provides cursor-paginated results, allowing users to browse past evaluations with full details.
  • evaluationTrend (query): Aggregates daily averages for each test type over the last 90 days, perfect for visualizing performance trends.

4. The Dashboard: Next.js & Recharts

The frontend, built with Next.js, brings all this data to life in src/app/(dashboard)/dashboard/personas/[id]/evaluations/page.tsx.

  • A clean header with the persona's avatar and a Radix DropdownMenu for selecting and running tests.
  • A stunning Recharts AreaChart displays the 90-day trend, with color-coded lines for temperature, jailbreak, and degradation scores. This visual feedback is invaluable for quickly spotting performance shifts.
  • An expandable results table provides granular detail for each evaluation run: individual scores, a list of detected violations, a checklist of present markers, and the full prompt/response pairs for deep dives.
  • Intuitive score badges (green >=80, yellow >=60, red <60) offer immediate visual cues about performance.

We also added a quick "Evaluations" button to the persona's main page (src/app/(dashboard)/dashboard/personas/[id]/page.tsx), making it easy to jump directly into the evaluation dashboard.

Lessons from the Trenches: Overcoming Development Hurdles

No project is without its quirks. Here are a couple of notable "pains" that turned into valuable lessons:

1. Component API Nuances: variant="destructive" vs. variant="danger"

The Problem: I instinctively reached for variant="destructive" on our internal Badge component to indicate a low score, but TypeScript immediately flagged it: TS2322: Type '"destructive"' is not assignable to type '"default" | "success" | "accent" | "warning" | "danger"'.

The Lesson: Always double-check your component library's API! While "destructive" might feel semantically correct in some contexts, our nyxcore Badge API explicitly uses "danger". It's a small detail, but knowing your tools prevents friction and ensures consistent UI. A quick switch to variant="danger" resolved it.

2. Type Safety with Prisma JSON Fields

The Problem: When dealing with Json fields in Prisma, especially when they might be empty or null, it's tempting to cast them carelessly. I initially tried as unknown as undefined for some optional JSON fields.

The Lesson: This was quickly (and rightly!) flagged in code review as a type-safety antipattern. Lying to the compiler about types can lead to subtle runtime bugs. The correct, semantically sound approach for Json fields in Prisma is to use as unknown as Prisma.InputJsonValue. This cast explicitly acknowledges that the value is a JSON-compatible type, allowing the compiler to correctly infer its structure without sacrificing type safety. This reinforces the value of thorough code reviews and adhering to best practices for robust codebases.

What's Next? Pushing the Boundaries

With the system in place, our immediate next steps involve:

  • Deployment & RLS Application: Committing the code, pushing it, and applying the RLS policies to the running database.
  • Manual Verification: Running initial jailbreak and temperature tests to confirm everything works as expected, particularly verifying roleAdherence and score progression in the trend chart.
  • Performance Optimization: One runAllTests can involve 14 LLM calls, potentially taking 70-210 seconds. We're already considering Server-Sent Events (SSE) for streaming updates to the UI, improving the user experience during long-running evaluations.
  • E2E Testing: Integrating end-to-end tests to ensure the evaluations page loads correctly and the "run test" interaction is seamless.

This Persona Evaluation & Benchmark System is a significant step forward in ensuring the reliability, consistency, and safety of our production LLM personas. It empowers us to catch regressions early, understand persona behavior deeply, and ultimately deliver more trustworthy AI experiences.

json
{"thingsDone":["Implemented PersonaEvaluation model with RLS","Developed persona-evaluator service with temperature, jailbreak, and degradation tests","Integrated LLM-as-Judge scoring with persona-specific markers","Created tRPC endpoints for evaluation, history, and trends","Built Next.js UI with Recharts trend chart and detailed results table","Applied code review fixes for RLS, tenant isolation, and type safety"],"pains":["Incorrect component variant usage (destructive vs. danger)","Type-safety antipattern with Prisma Json fields (as unknown as undefined)"],"successes":["Achieved feature completeness for LLM persona evaluation","Implemented robust RLS for multi-tenancy","Successfully used LLM-as-Judge for objective scoring","Created intuitive trend visualization with Recharts","Ensured type-clean and lint-clean codebase"],"techStack":["TypeScript","Next.js","Prisma","tRPC","Recharts","PostgreSQL","LLMs (implicit)","Radix UI"]}