← nyxcore-systems
7 min read

Late Night Wins: Conquering SSE, Persistent Reports, and Schema Headaches

Another late-night coding session, another set of challenges tackled! This recap dives into fixing critical SSE reconnects, building robust report persistence with PDF export, and navigating some tricky Prisma schema migrations.

TypeScriptNext.jsPrismaSSEPDFDatabaseBugFixDeveloperExperienceWorkflowEngine

Another late night, another intense coding session wrapped up. As the clock ticked past midnight, the satisfaction of pushing a significant chunk of work felt pretty good. This session was all about stability, user experience, and laying down crucial infrastructure for future features. We tackled everything from fixing elusive SSE reconnect issues to making reports truly persistent and exportable.

Let's dive into the trenches and see what got done, what lessons were learned, and what's next on the horizon.

The Mission Brief: What We Set Out to Conquer

The primary goals for this session were ambitious:

  1. Stabilize SSE Reconnects: Ensure live updates for auto-fix and refactor workflows don't break when users navigate away and return.
  2. Smarter Workflows: Automatically inject context into custom workflow prompts, reducing manual effort.
  3. Robust Reporting: Implement full report persistence to the database and enable PDF exports.
  4. UI Polish: Enhance report modal headers for better context.

Spoiler alert: all objectives were met, pushed, and are now live!

Feature Highlights: The Wins We're Celebrating

1. Auto-Context Injection: Supercharging Custom Workflows

One of our goals is to make custom workflows as intelligent and effortless as possible. Previously, users might have had to manually reference preceding steps to build complex prompts. Not anymore.

We've introduced buildAutoContext() in src/server/services/workflow-engine.ts. This function now intelligently scans for available context and, crucially, a fallback in resolvePrompt() ensures that if no explicit {{steps.*}} references are found, relevant context is still injected. This means less boilerplate for developers building workflows and more powerful, dynamic prompts.

typescript
// Conceptual snippet illustrating auto-context in workflow-engine.ts
async function resolvePrompt(promptTemplate: string, context: WorkflowContext): Promise<string> {
  // ... existing step reference resolution ...

  if (!promptTemplate.includes('{{steps.')) {
    // If no explicit step references, attempt to auto-inject relevant context
    const autoContext = buildAutoContext(context);
    return renderTemplate(promptTemplate, { ...context, autoContext });
  }

  return renderTemplate(promptTemplate, context);
}

(Commit: 243cadd)

2. SSE Reconnect Fix: No More Lost Updates!

This was a critical one. Users interacting with our auto-fix and refactor features were experiencing a frustrating bug: if they switched tabs or navigated away from the detail page and then returned, the live updates from our Server-Sent Events (SSE) would simply stop. The progress bar would freeze, and they'd lose visibility into the ongoing process.

The Root Cause: Our SSE endpoints (/events/auto-fix/[id] and /events/refactor/[id]) were sending a one-shot "refresh" message and then closing the connection for any run that wasn't strictly "pending." The problem? The backend pipeline for these runs continues server-side even after a disconnect. When the user reconnected, the SSE endpoint would hit the status !== "pending" guard, assume the run was finished (even if it wasn't), and promptly close the connection again. Dead end.

The Fix: We've refactored these endpoints to actively poll the database every 3 seconds for active runs. For runs that reach a terminal state (succeeded, failed), we now explicitly send a done event before closing the connection. This ensures a robust, continuous stream of updates, even across reconnections. Our workflows and code-analysis features didn't suffer from this thanks to a different underlying architecture, but now auto-fix and refactor are just as resilient.

(Commit: d74b32a)

3. Persistent Reports & PDF Export: Audit Trails and Shareability

This was a big one for product value. Previously, generated reports were ephemeral. Now, they're first-class citizens:

  • Report Persistence: We introduced a new Report Prisma model, complete with Row-Level Security (RLS) for multi-tenancy. Our src/server/trpc/routers/reports.ts now handles listing, getting, and deleting reports. All three generateReport mutations now persist their output directly to the database.
  • PDF Export: To make these reports truly useful, we added a /api/v1/reports/pdf POST endpoint. This leverages a Python script (scripts/md2pdf.py) that uses md2pdf-mermaid and Playwright to convert markdown reports into beautiful, shareable PDFs. Users can now download both markdown and PDF versions directly from the ReportGeneratorModal and the ReportsTab viewer Sheet.
python
# scripts/md2pdf.py (Conceptual snippet)
import markdown_to_pdf.markdown_to_pdf as md2pdf
from playwright.sync_api import sync_playwright

def convert_md_to_pdf(markdown_content: str, output_path: str):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        # md2pdf_mermaid handles the conversion and rendering with Playwright
        md2pdf.convert_md_to_pdf(
            markdown_string=markdown_content,
            output_file=output_path,
            playwright_page=page
        )
        browser.close()

(Commit: d123ccf)

4. UI Polish & Infrastructure Tweaks

  • Contextual Report Headers: The ReportGeneratorModal header now dynamically displays projectName + reportType + date instead of a static "nyxCore," providing immediate context. (Commit: d74b32a)
  • QR Code Update: A minor but important detail, the QR code URL in src/server/services/report-generator.ts:72 was updated to nyxcore.cloud. (Commit: 243cadd)
  • Git Ignore: Added .venv/ to .gitignore to keep things clean.

Lessons Learned: The "Pain" Log Transformed

Not every path is smooth. Here were a couple of critical "gotchas" that taught us valuable lessons:

1. Prisma & Custom Database Types: The vector(1536) Saga

The Problem: After adding the new Report model, I tried to push the schema changes with prisma db push --accept-data-loss. To my dismay, Prisma dropped our critical embedding vector(1536) column on the workflow_insights table. Prisma, being an ORM, doesn't natively understand custom PostGIS vector types, treating them as unsupported during schema migrations.

The Insight: While ORMs simplify database interactions, they can sometimes be too opinionated or lack support for highly specialized database features. When working with custom types or extensions, always be aware of your ORM's limitations during schema migrations.

Actionable Takeaway: For now, the workaround is to manually restore the column and recreate its HNSW index after every db:push. This is a recurring issue, so we're looking into more robust migration strategies for custom types, perhaps using raw SQL migrations or a more nuanced ALTER TABLE approach within a custom migration script.

sql
-- The essential workaround after a Prisma db push
ALTER TABLE workflow_insights ADD COLUMN IF NOT EXISTS embedding vector(1536);
-- Recreate HNSW index if dropped
-- CREATE INDEX workflow_insights_embedding_hnsw ON workflow_insights USING hnsw (embedding vector_l2_ops);

2. The Elusive Cannot read properties of undefined (reading 'create')

The Problem: After adding the Report model to the Prisma schema, generating a report immediately threw a Cannot read properties of undefined (reading 'create') error at auto-fix.ts:480. This was puzzling at first, as the schema looked correct.

The Insight: This is a classic "did you regenerate your client?" moment. Adding a new model to schema.prisma requires the Prisma client to be regenerated so that the TypeScript types and runtime methods (like prisma.report.create()) are available. Without regeneration, the old client doesn't know about the new model.

Actionable Takeaway: After any change to schema.prisma, always run npm run db:generate. It's a fundamental step in the Prisma workflow. Our dev-start.sh script already includes this, but it's easy to forget when manually pushing schema changes. This serves as a good reminder to always include db:generate in any schema-altering script or command sequence.

What's Next on the Horizon?

With these critical updates deployed, our focus shifts to enhancing user experience and expanding capabilities:

  1. Persona Chooser Refactor: A unified <PersonaPicker> component is coming. Think collapsible sections, rich persona cards with portraits, traits, specializations, and levels. This will replace all existing persona selectors across the application, standardizing how users interact with our AI agents. The goal is a seamless expert team phd → discuss → best approach → implement pattern.
  2. Report Tab Styling: Making the Reports tab in the project detail page visually stunning while adhering to the nyx design system.
  3. Custom Workflow Testing: Thoroughly test the new auto-context injection with complex custom workflows (analyse → review → generate prompt).
  4. RAG-based Policy Library: Exploring a RAG (Retrieval Augmented Generation)-based policy library per tenant for compliance standards (DSGVO, ISO 27001), moving beyond pure LLM approaches for accuracy and relevance.

It was a productive session, reinforcing the importance of robust error handling, diligent schema management, and a relentless focus on user experience. Stay tuned for more updates as we continue to build out nyxCore!

json
{
  "thingsDone": [
    "Auto-context injection in workflow engine",
    "SSE reconnect fix for auto-fix/refactor live updates",
    "Report persistence with new Prisma model and RLS",
    "PDF export functionality for reports (md2pdf-mermaid, Playwright)",
    "Report modal header improvements for context",
    "QR code URL updated to nyxcore.cloud",
    "Prisma client regeneration after schema changes",
    ".venv/ added to .gitignore"
  ],
  "pains": [
    "Prisma db push dropping custom vector(1536) column, requiring manual restore",
    "Prisma client not updated after schema changes causing 'create' method undefined error",
    "SSE endpoints closing connection prematurely for active, non-pending runs"
  ],
  "successes": [
    "Resolved critical SSE live update issue",
    "Implemented full report persistence and export, enhancing data utility",
    "Improved developer workflow with auto-context injection",
    "Learned and documented key Prisma migration workarounds"
  ],
  "techStack": [
    "TypeScript",
    "Next.js",
    "Prisma",
    "PostgreSQL",
    "Server-Sent Events (SSE)",
    "Python",
    "md2pdf-mermaid",
    "Playwright",
    "TRPC"
  ]
}