← nyxcore-systems
9 min read

Untitled

title: Shipping Persona Evaluation v2 - A Late Night Odyssey into Hybrid AI Scoring and Profile Management date: 2026-03-12 tags:

  • AI
  • LLM
  • Prisma
  • TypeScript
  • tRPC
  • Deployment
  • Database Migration
  • Persona
  • Hybrid Scoring
  • Engineering excerpt: Dive into the intense late-night session where we successfully rolled out Persona Evaluation v2, introducing profile-driven hybrid scoring, deterministic evaluations, and a robust profile management system. Discover the technical challenges we overcame, from schema migrations to deployment verification, and the innovative solutions that brought this complex feature to life.

It was late. The coffee was cold, the clock ticked past midnight, but the mission was crystal clear: ship Persona Evaluation v2. This wasn't just a minor update; it was a complete overhaul, an ambitious sprint encompassing 11 distinct tasks across four major chunks, from foundational type definitions all the way to production deployment.

And I'm thrilled to report: mission accomplished. Every single task, every line of code, every database migration, merged to main and running live.

The Why: Beyond Generic LLM Evaluation

Our existing persona evaluation system, while functional, relied heavily on generic LLM prompts for scoring. This left a crucial gap: how do you consistently measure adherence to a specific persona's nuanced traits, detect subtle jailbreaks, or quantify a model's "refusal quality" without a tailored rubric? The answer lay in profile-driven, hybrid scoring – a blend of deterministic checks and LLM advisory.

This journey began with a vision:

  1. Define the Language: Create a robust type system for every facet of a persona and its evaluation.
  2. Schema Evolution: Adapt our database to store this rich, new data.
  3. Built-in Intelligence: Equip our system with a library of predefined persona profiles.
  4. The New Brain: Rewrite the core evaluation logic to use these profiles for hybrid, deterministic scoring.
  5. User Empowerment: Build UI and API tools for managing these profiles and visualizing the new evaluation metrics.
  6. Seamless Launch: Deploy it all to production.

Let's dive into the how.

Chunk 1: Laying the Foundation – Types, Schema & Profiles

Before we could build the engine, we needed the blueprint. This meant meticulously defining every data structure: PersonaProfile, AttackVector, MarkerDefinition, JudgeInput, JudgeOutput, Violation, MarkerResult, WeightProfile. We even added a WEIGHT_PROFILES constant to manage scoring sensitivities. Crucial utility functions like computeCompositeScore(), validateProfile(), and checkDiscrepancy() were born here.

Our prisma/schema.prisma underwent a significant transformation, gaining 7 new columns on the PersonaEvaluation model (think refusalQuality, coherenceUnderLoad, compositeScore, discrepancyFlag) and an entirely new PersonaProfile model to store persona-specific data.

But the real game-changer in this chunk was src/server/services/persona-profiles.ts. This file became home to 12 meticulously crafted built-in persona profiles – from the stoic Cael to the trickster Hermes, the wise Athena to the disruptive Nemesis. Each profile defined its unique traits, scoring weights, and even specific attack vectors. We also added a GENERIC_PROFILE fallback and a compile-time check to ensure profile integrity.

Chunk 2: The Brain Transplant – Deterministic Scoring & Evaluator Rewrite

This was the heart of Persona Evaluation v2. The src/server/services/persona-evaluator.ts file was completely rewritten, shedding its generic skin for a sophisticated, profile-driven hybrid approach.

Here's a glimpse of the new intelligence:

  • scoreMarkersDeterministic(): No more guessing. This regex-based system provides tamper-proof, objective scoring for specific markers.
  • scoreRoleAdherenceDeterministic(): We moved beyond simple keyword matching. This function now uses behavior matching and anti-pattern penalties to robustly assess how well an LLM adheres to its persona.
  • buildJudgePrompt(): Instead of a one-size-fits-all prompt, the LLM judge now receives highly persona-specific instructions, guiding its qualitative assessment with far greater precision.
  • hybridScore(): The star of the show. This function intelligently combines the deterministic primary score with an LLM advisory score, crucially featuring discrepancy detection to flag when the two diverge significantly.
  • loadProfile(): A smart loader that prioritizes built-in profiles, then approved database profiles, before falling back to the generic profile.
  • persistEval(): Now stores all the rich v2 fields like compositeScore, judgeReasoning, attackVector, evalTier, and discrepancyFlag.
  • Tiered Evaluations: We introduced runQuickEval() (temperature test + 3 generic jailbreaks) and runFullEval() (all 3 test types + tailored attack vectors from the persona profile). This allows for rapid checks and deep dives.

Chunk 3: Bringing it to Life – Profile Management & UI

What's powerful backend logic without an intuitive frontend? We made Persona Evaluation v2 accessible and manageable.

Our src/server/trpc/routers/personas.ts gained new superpowers:

  • generateProfileDraft: Imagine this – an LLM-powered procedure that can derive a profile draft directly from a persona's system prompt!
  • approveProfile: A critical step for curating and validating persona profiles.
  • getProfile: To fetch profile data for display and editing.
  • runEvaluation: Updated to support the new "quick" and "full" evaluation tiers.

On the UI front, the changes were dramatic:

  • dashboard/personas/[id]/evaluations/page.tsx: Now proudly displays the compositeScore, complete with attack vector, tier, and discrepancy badges. Bonus dimensions like refusalQuality and coherenceUnderLoad are visible, alongside a dedicated section for judgeReasoning. A new dropdown allows users to select "Quick Eval" or "Full Eval."
  • dashboard/personas/[id]/profile/page.tsx: A brand new dedicated profile management page! It features status badges (BUILT-IN/DRAFT/APPROVED/NO PROFILE), a JSON viewer/editor, and buttons to "Generate Draft," "Approve," or "Regenerate" profiles.

Chunk 4: The Finish Line – Deployment

The final hurdle: getting it all into production. This involved a direct SQL schema migration (more on the "pain" of that in a moment!) and a full rebuild and deployment of the application.

The Gauntlet: Navigating the "Pain" Log (CRITICAL Lessons)

No complex deployment is without its battles. This session was no exception, yielding some critical lessons:

  1. Database Migration Mayhem:

    • Tried: Our trusty ./scripts/db-migrate-safe.sh --dry-run on production.
    • Failed: ERROR: DATABASE_URL not set. Our script couldn't read .env.production vars, and even with manual env setting, npx wasn't available on the host.
    • The Shock: When we finally generated the migration diff inside the app container, it included dangerous DROP commands for unrelated tables! (e.g., embedding columns, crawl_jobs table).
    • Workaround & Lesson: We abandoned the script. Instead, we manually extracted only our ALTER TABLE and CREATE TABLE statements. Then, we applied them directly via SQL: docker exec nyxcore-postgres-1 psql -U nyxcore -d nyxcore -c "ALTER TABLE ... / CREATE TABLE ..."
    • Key Takeaway: For targeted schema changes on production, especially in a Dockerized environment, always use direct SQL to avoid unintended consequences and ensure precision.

  2. Prisma Client Regeneration Blues:

    • Tried: Deploying the new code without explicitly regenerating the Prisma client locally against a new schema.
    • Failed: A cascade of TypeScript errors erupted. The locally generated Prisma client didn't know about the new columns and models.
    • Workaround & Lesson: A quick npx prisma generate resolved all type errors. Crucially, we remembered that npx prisma generate only needs the schema.prisma file, not an active database connection, making it easy to run locally.

  3. Deployment Verification Blind Spots:

    • Tried: curl localhost:3000 from the production host to verify the app was running.
    • Failed: Returned 000. Our app is behind Nginx, not directly accessible from the host network.
    • Workaround & Lesson: To verify internal service health, we had to go inside the Docker network: docker exec nyxcore-nginx-1 wget -q -O /dev/null -S http://app:3000/. This confirmed the app container was responsive to Nginx.

The Triumph

Despite the hurdles, the session concluded with a resounding success. All 11 tasks are complete. The feat/persona-eval-v2 branch is merged. The schema is migrated. The application is deployed and running on production, delivering a far more intelligent, robust, and user-friendly persona evaluation experience.

The system now features:

  • Deterministic, profile-driven scoring for objective measurements.
  • Hybrid evaluation combining deterministic checks with nuanced LLM judgment.
  • Discrepancy detection to highlight interesting edge cases.
  • Tiered evaluations for quick checks and deep dives.
  • A powerful profile management UI with LLM-assisted draft generation.

This wasn't just about shipping a feature; it was about elevating our entire approach to AI persona quality assurance.

What's Next? (The Immediate Horizon)

While the main mission is complete, a few immediate verifications and cleanups remain:

  1. Production Verification: Confirm Cael's evaluations page shows v2 scoring, run a Quick Eval, check the profile page, and run a Full Eval to test tailored attack vectors.
  2. Security Enhancement: Add RLS policies for the persona_profiles table (currently protected by tRPC middleware only).
  3. Custom Persona Flow: Test the end-to-end flow for creating a custom persona, generating a draft, approving it, and running an evaluation.
  4. Clean Up: Delete the now-merged feature branch.

The journey continues, but tonight, we celebrate a significant leap forward.

json
{
  "thingsDone": [
    "Implemented comprehensive type definitions for persona evaluation (PersonaProfile, AttackVector, MarkerDefinition, JudgeInput, JudgeOutput, Violation, MarkerResult, WeightProfile).",
    "Extended Prisma schema with 7 new columns for PersonaEvaluation and added a new PersonaProfile model.",
    "Created 12 built-in persona profiles and a generic fallback, with compile-time checks.",
    "Rewrote the persona evaluation service to support profile-driven hybrid scoring (deterministic primary + LLM advisory).",
    "Developed deterministic scoring mechanisms for markers (regex-based) and role adherence (behavior matching + anti-pattern penalty).",
    "Implemented persona-specific LLM judge prompt generation and discrepancy detection.",
    "Introduced tiered evaluations (Quick Eval, Full Eval) with generic and profile-specific attack vectors.",
    "Added tRPC procedures for generating profile drafts (LLM-derived), approving profiles, and retrieving them.",
    "Updated the evaluation UI to display v2 fields (composite score, attack vector, tier, discrepancy, judge reasoning, bonus dimensions).",
    "Developed a dedicated profile management UI with status badges, JSON viewer/editor, and draft/approve workflow.",
    "Successfully performed schema migration and deployed the application to production."
  ],
  "pains": [
    "Database migration script (`db-migrate-safe.sh`) failed due to environment variable issues and `npx` unavailability on the production host.",
    "Encountered dangerous `DROP` commands in auto-generated Prisma migration diffs, necessitating manual SQL application.",
    "TypeScript errors cascaded due to an outdated Prisma client, requiring `npx prisma generate` without a database connection.",
    "Initial deployment verification via `curl localhost:3000` failed due to the app being behind Nginx, requiring an internal Docker network check."
  ],
  "successes": [
    "Successfully implemented a robust, profile-driven hybrid AI scoring system.",
    "Achieved deterministic evaluation capabilities for key metrics.",
    "Developed an intuitive profile management system, including LLM-powered draft generation.",
    "Overcame complex database migration challenges through targeted SQL.",
    "Ensured all 11 planned tasks for Persona Evaluation v2 were completed and deployed within a single session.",
    "Gained valuable insights into production deployment and migration best practices in a Dockerized environment."
  ],
  "techStack": [
    "TypeScript",
    "Prisma",
    "tRPC",
    "Next.js",
    "React",
    "PostgreSQL",
    "Docker",
    "Nginx",
    "Large Language Models (LLMs)"
  ]
}