← nyxcore-systems
6 min read

Beyond the Prompt: Building Smarter LLM Dev Tools with User Control and Self-Learning Loops

A deep dive into enhancing our AI-powered dev tools, from giving users control over LLM providers to laying the groundwork for a self-improving learning system.

LLMsDeveloperToolsFrontendBackendTypeScriptPrismaFeedbackLoopsAIEngineering

Building intelligent developer tools powered by Large Language Models (LLMs) is a fascinating journey. It's a blend of cutting-edge AI, robust software engineering, and a constant focus on the developer experience. In my latest session, I tackled two critical areas: empowering users with more control and kickstarting our ambitious closed-loop learning system.

Empowering the User: Choosing Your AI Brain

One of the first things you realize when integrating LLMs into a product is that "LLM" isn't a monolith. There are many providers, many models, each with its strengths, weaknesses, and cost implications. For a tool like our AutoFix and Refactor features, giving users the choice of which LLM powers their operations isn't just a nice-to-have; it's essential for flexibility and performance tuning.

This session's first major win was integrating a provider/model chooser directly into our AutoFix and Refactor dialogs.

The Implementation Details:

  • UI Integration: I added a LLM_PROVIDERS button group and a corresponding model input to both auto-fix/page.tsx and refactor/page.tsx. This gives immediate control where it matters most.
  • Visibility: To ensure transparency and traceability, the selected provider and model now appear as badges in the detail headers (auto-fix/[id]/page.tsx and refactor/[id]/page.tsx) and as labels on the run list cards. This makes it easy to see at a glance which AI 'brain' was used for a particular operation.
  • Backend Handshake: The frontend now passes the chosen provider and model directly to our tRPC start mutations. Thankfully, the backend was already set up to accept these parameters, making the integration smooth.

This feature might seem small, but it's a huge step towards making our tools more adaptable and user-centric. It acknowledges the diverse needs and preferences of developers working with different LLM ecosystems.

The Dance of State and Reality: A Frontend Tale

Even with exciting new features, the life of a developer is often about wrestling with existing code and squashing pesky bugs. This session had its share, specifically a subtle but critical phase synchronization issue.

The Problem: When UI Lies to You

Our AutoFix and Refactor processes go through several phases (e.g., "scanning," "detecting," "improving"). The UI displays the currentPhase of an active run. Initially, I had hardcoded useState<RefactorPhase>("scan") as the initial phase when a user navigated to a run's detail page.

The problem? If a user navigated away from an active run (say, to another page) and then returned, the UI would incorrectly show "scanning" or "detecting" even if the backend process had moved on to "improving." The local frontend state was out of sync with the true state stored in our database.

The Fix: Trusting the Source of Truth

The solution, while straightforward in hindsight, required a clear understanding of state management principles.

Instead of relying solely on local useState for the initial phase, I introduced a useEffect hook on both [id]/page.tsx detail pages. This useEffect now reads the run.status from our database query on component mount (or when run.status changes) and maps it through a statusToPhase utility to set the correct currentPhase.

typescript
// Inside auto-fix/[id]/page.tsx or refactor/[id]/page.tsx
// Assuming 'run' is fetched from tRPC query
import { useEffect, useState } from 'react';
import { statusToPhase, RefactorPhase } from '@/utils/status-mapping'; // Example utility

// ... component body ...
const [currentPhase, setCurrentPhase] = useState<RefactorPhase>("scan"); // Initial default

useEffect(() => {
  if (run?.status) {
    // Sync currentPhase from the backend's source of truth
    setCurrentPhase(statusToPhase[run.status]);
  }
}, [run?.status]); // Re-run effect when the run's status from the DB changes

// Note: SSE events will continue to update currentPhase dynamically as they arrive,
// ensuring real-time updates after the initial sync.

This ensures that the UI always reflects the actual state of the run as recorded in the database, preventing user confusion. Real-time updates via Server-Sent Events (SSE) then take over to provide a smooth, live experience.

Lesson Learned: The Primacy of the Backend

Takeaway: Never let ephemeral frontend state completely override the authoritative source of truth, especially for critical process states. Use useEffect to bridge the gap between backend data and local UI state, ensuring consistency and reliability.

Another Minor Quirk: Prisma's Json? Field

While not a bug, dealing with Prisma's Json? type for our config field presented a minor annoyance. Accessing nested properties often requires explicit type casting, like as Record<string, string> | null or even as unknown as in some contexts. It highlights the trade-offs of using schemaless JSON fields in a strongly typed TypeScript environment. It works, but it adds a bit of boilerplate.

Towards a Self-Improving Future: The Learning Loop

Perhaps the most exciting development from this session is the official kickoff of our "closed-loop learning system." This is where we transcend simple LLM integration and move towards building truly intelligent, self-improving systems.

The Vision: AI That Learns from Experience

The core idea is simple yet powerful: pipeline findings (e.g., successful fixes, failed refactors, user feedback) should feed back into the system's "memory" or "knowledge base." This knowledge can then be used to inform and improve future LLM prompts, leading to better, more contextual, and more reliable outcomes over time.

Laying the Groundwork: Research and Architecture

This is a significant undertaking, so we've created a dedicated learning-loop team. Our immediate next steps are heavily focused on research and architectural design:

  1. Insight/Memory Architecture: How do we represent and store the "learnings"? What kind of data structure is best suited for this? How do we efficiently retrieve relevant insights?
  2. Pipeline Data Flow: What data points from our existing AutoFix/Refactor pipelines are most valuable for learning? How do we extract them cleanly?
  3. Designing the Feedback Loop: This involves defining the overall system architecture – how insights are processed, stored, and made available.
  4. Implementing Insight Extraction: Building the actual logic to pull meaningful "lessons" from pipeline results.
  5. Implementing Insight Injection: Figuring out how to dynamically inject these extracted insights into subsequent LLM prompts to guide their behavior.
  6. End-to-End Verification: Rigorous testing to ensure the entire system works as intended and actually improves performance.

We're currently spawning research agents for tasks 1 and 2, which will unblock the subsequent design and implementation phases. This is where the real innovation happens – moving from reactive fixes to proactive learning.

What's Next?

With the provider chooser pushed, the phase sync bug committed, and the learning-loop team actively researching, we're setting the stage for some profound enhancements. The immediate focus is on solidifying the foundational research for our learning system, paving the way for an AI assistant that truly evolves and gets smarter with every interaction.

It's a challenging but incredibly rewarding path, building tools that don't just use AI, but embody intelligence.

json
{
  "thingsDone": [
    "Added LLM provider and model chooser to AutoFix and Refactor dialogs",
    "Displayed chosen provider/model on detail pages and run list cards",
    "Integrated provider/model selection with tRPC start mutations",
    "Fixed a critical UI phase desynchronization bug on detail pages",
    "Created dedicated 'learning-loop' team and initiated research for closed-loop learning system"
  ],
  "pains": [
    "Initial UI state desync due to hardcoded phase, leading to incorrect user experience",
    "Required explicit type casting for Prisma's `Json?` fields, adding boilerplate"
  ],
  "successes": [
    "Successfully empowered users with LLM provider/model choice, enhancing flexibility",
    "Implemented a robust fix for UI state desynchronization using `useEffect` and backend source of truth",
    "Successfully launched the research phase for a groundbreaking self-improving AI learning system"
  ],
  "techStack": [
    "Next.js",
    "React",
    "TypeScript",
    "tRPC",
    "Prisma",
    "LLMs (various providers)",
    "Frontend Development",
    "Backend Development",
    "State Management",
    "AI/ML Engineering"
  ]
}