← nyxcore-systems
8 min read

Unlocking LLM Flexibility: Our Journey to Dynamic Provider & Model Selection

Dive into how we engineered a robust system for dynamic LLM provider and model selection, enabling tenant defaults, mid-discussion switching, and graceful error recovery in our AI-powered application.

LLMAITypeScripttRPCPrismaFrontendBackendUXDeveloperExperienceNext.js

Building AI-powered applications is exciting, but it quickly brings you face-to-face with a fundamental challenge: the ever-evolving LLM landscape. Different providers, countless models, varying costs, speeds, and capabilities – how do you offer users the flexibility to choose, provide sensible defaults, and ensure a resilient experience when things go sideways? This was the core problem we tackled in our latest development sprint.

Our goal was ambitious: implement a smart LLM provider and model selection system for discussions. This meant handling tenant-level defaults, allowing users to switch models mid-discussion, providing graceful fallback UX during errors, and maintaining a clear, extensible model catalog. After a focused session, I'm thrilled to report that this feature is fully implemented and committed. Let's break down the journey, the technical decisions, and a key lesson learned along the way.

The Challenge: Navigating the LLM Maze

Imagine an application where users can have AI-powered discussions. Initially, we might hardcode a single provider and model. But as soon as you think about:

  • Cost optimization: Switching to a cheaper model for less critical tasks.
  • Performance: Using a faster model when latency is paramount.
  • Capabilities: Leveraging a specific model's strength (e.g., code generation vs. creative writing).
  • Redundancy: What if OpenAI goes down? Can we switch to Anthropic?
  • Tenant-specific needs: Different organizations might prefer different LLMs by default.

Suddenly, hardcoding becomes a nightmare. We needed a system that was dynamic, user-friendly, and robust.

Our Solution: A Layered Approach to LLM Selection

We approached this by building a layered system, from database schema to frontend components, all orchestrated by tRPC.

1. The Foundation: Database Schema & Defaults

The journey began with our Prisma schema. We needed to store preferences:

  • Tenant Defaults: Each Tenant can now specify a defaultProvider and defaultModel. This is crucial for setting sensible starting points across an organization.
    prisma
    // prisma/schema.prisma
model Tenant {
  id             String @id @default(uuid())
  name           String
  defaultProvider String?
  defaultModel   String?
  // ... other fields
}
  • Discussion Overrides: A specific discussion might need to deviate from the tenant's default. We added model_override to the Discussion model. This allows a discussion to "lock in" a specific model for its lifetime, overriding any defaults.
    prisma
    // prisma/schema.prisma
model Discussion {
  id              String @id @default(uuid())
  title           String
  tenantId        String
  model_override  String? // Nullable, so existing discussions don't break
  // ... other fields
}

After these changes, a quick npx prisma db push and npx prisma generate updated our database and regenerated the Prisma client, keeping everything in sync.

2. The LLM Catalog: Our Source of Truth

To manage the diverse LLM landscape, we created a static MODEL_CATALOG in src/lib/constants.ts. This catalog holds essential information about each model: its provider, display name, ID, and crucial hints like cost and speed.

typescript
// src/lib/constants.ts (simplified)
export interface ModelInfo {
  id: string;
  provider: string;
  name: string;
  costHint: 'low' | 'medium' | 'high';
  speedHint: 'fast' | 'medium' | 'slow';
  bestFor?: string;
  // ... other metadata
}

export const MODEL_CATALOG: ModelInfo[] = [
  { id: 'gpt-4o', provider: 'openai', name: 'GPT-4o', costHint: 'high', speedHint: 'fast', bestFor: 'Complex tasks' },
  { id: 'claude-3-opus-20240229', provider: 'anthropic', name: 'Claude 3 Opus', costHint: 'high', speedHint: 'medium', bestFor: 'Creative writing' },
  { id: 'gemini-1.5-pro', provider: 'google', name: 'Gemini 1.5 Pro', costHint: 'medium', speedHint: 'medium', bestFor: 'Multi-modal' },
  { id: 'kimi', provider: 'kimi', name: 'Kimi Chat', costHint: 'low', speedHint: 'fast', bestFor: 'Long context' },
  // ... more models
];

// Helper functions to query the catalog
export const getModelsForProvider = (providerId: string) => ...;
export const getDefaultModel = (providerId: string) => ...;
export const getModelInfo = (modelId: string) => ...;

This static approach keeps things simple for now. Adding a new model is a direct code change, which is acceptable at this stage.

3. API & Backend Logic: tRPC & discussion-service

Our tRPC API was extended to support the new functionality:

  • discussions.availableProviders: A query to check which LLM providers have API keys configured for the current tenant. This determines what options are even possible for the user.
  • discussions.updateProvider / discussions.updateModel: Mutations allowing users to change the LLM mid-discussion.
  • discussions.create: Now accepts an optional modelOverride to kick off a new discussion with a specific LLM.
  • admin.getDefaults / admin.updateDefaults: Queries/mutations for administrators to manage tenant-wide LLM defaults. Validation and audit logging are built into updateDefaults for security and traceability. (Note: getDefaults uses enforceTenant so any authenticated user can read them, but only admins can write.)

On the backend, our discussion-service.ts was updated. All four discussion modes (single, parallel, consensus, autoRound) now respect the model_override when making calls to the underlying LLM provider, passing it as part of the LLMCompletionOptions. This ensures that the selected model is consistently used throughout the discussion flow.

4. The Frontend: UX & The ProviderPicker

This is where all the backend work comes to life for the user.

  • New Discussion Page (new/page.tsx): We integrated the ProviderPicker component here. It intelligently pre-selects the tenant's default provider and model, shows which providers are available (based on API keys), and displays the model selector with our cost, speed, and "best for" hints. This provides a clear starting point.

  • Discussion Detail Page ([id]/page.tsx):

    • Mid-Discussion Switching: We added clickable provider labels within our StreamFlow UI. Clicking these opens the ProviderPicker, allowing users to dynamically switch the LLM for subsequent messages.
    • Error Recovery: If an LLM call fails (e.g., due to an invalid API key or a model being unavailable), an inline error retry UI appears. This UI includes the ProviderPicker, allowing the user to select a different provider/model, alongside a "Retry same" button for transient errors. This significantly improves resilience.

  • Admin Page: A new "LLM Defaults" tab provides an intuitive interface for administrators to set and save the defaultProvider and defaultModel for their tenant using selection cards.

Lessons Learned: The ProviderPicker Double-Click Dilemma

One particular UI challenge stood out during the development of the ProviderPicker component (src/components/discussion/provider-picker.tsx).

The Problem: Initially, I tried to render the ProviderPicker component conditionally on the discussion detail page, wrapped in a showProviderPicker state. The idea was: click a button -> showProviderPicker becomes true -> ProviderPicker renders. However, because the ProviderPicker also managed its own internal isOpen state (triggered by a click on its internal toggle button), this created a frustrating double-click issue. The first click would render the component, but the user would then need a second click to actually open its internal dropdown.

The Fix: I added defaultOpen and onClose props to the ProviderPicker.

  • defaultOpen: boolean: If true, the component's internal dropdown opens immediately upon being rendered.
  • onClose: () => void: A callback that the ProviderPicker invokes when its internal click-outside detection closes the dropdown.

This allowed the parent component to control the initial visibility and be notified when the user was done interacting.

typescript
// Inside ProviderPicker.tsx (simplified)
import React, { useState, useEffect, useRef } from 'react';
import { useClickOutside } from '@mantine/hooks'; // Example hook for click-outside detection

interface ProviderPickerProps {
  defaultOpen?: boolean; // New prop
  onClose?: () => void;  // New prop
  // ... other props like currentProvider, onSelectModel, etc.
}

const ProviderPicker: React.FC<ProviderPickerProps> = ({ defaultOpen = false, onClose, ...props }) => {
  const [isOpen, setIsOpen] = useState(defaultOpen); // Initialize with defaultOpen
  const ref = useClickOutside(() => { // Detect clicks outside the component
    if (isOpen) {
      setIsOpen(false);
      onClose?.(); // Notify parent when closed by click-outside
    }
  });

  // Ensure internal state syncs with defaultOpen if it changes externally
  useEffect(() => {
    setIsOpen(defaultOpen);
  }, [defaultOpen]);

  const toggleOpen = () => {
    const newState = !isOpen;
    setIsOpen(newState);
    if (!newState) {
      onClose?.(); // Notify parent if closed by internal toggle
    }
  };

  return (
    <div ref={ref} className="relative">
      <button onClick={toggleOpen}>
        {/* Display current selection or "Choose LLM" */}
      </button>
      {isOpen && (
        <div className="absolute z-10 bg-white shadow-lg rounded-md mt-2">
          {/* Render provider and model options with cost/speed hints */}
        </div>
      )}
    </div>
  );
};

// In a parent component (e.g., discussion detail page)
{showProviderPicker && ( // Conditional rendering based on parent state
  <ProviderPicker
    defaultOpen={true} // Crucially, opens immediately on render
    onClose={() => setShowProviderPicker(false)} // Parent hides it when picker closes
    // ... other props
  />
)}

This pattern is incredibly useful for components that manage their own internal state but need to be programmatically controlled by a parent at specific times.

What's Next?

With the core functionality in place, our immediate next steps involve rigorous testing and future enhancements:

  1. End-to-end Testing: Verifying tenant defaults, new discussion pre-selection, error recovery flows, and mid-discussion switching.
  2. Expanding the Catalog: While static works for now, we'll consider adding more models like Ollama (for local LLM inference) to the MODEL_CATALOG.
  3. Persisting Model Choices: Currently, the chosen model is passed through for streaming. We should consider persisting the actual model used for each DiscussionMessage in the database, adding more granular historical data.

This sprint was a significant step towards a more flexible, resilient, and user-friendly AI experience. By abstracting away the complexities of LLM provider and model management, we empower both our users and administrators to tailor their AI interactions to their specific needs.

json
{
  "thingsDone": [
    "Added defaultProvider/defaultModel to Tenant model",
    "Added model_override to Discussion model",
    "Ran Prisma db:push and db:generate",
    "Created ModelInfo interface and MODEL_CATALOG with helpers",
    "Implemented discussions.availableProviders tRPC query",
    "Implemented discussions.updateProvider and discussions.updateModel tRPC mutations",
    "Extended discussions.create to accept modelOverride",
    "Added admin.getDefaults query and admin.updateDefaults mutation",
    "Updated discussion-service.ts to pass model_override to LLM calls",
    "Created reusable ProviderPicker component with grouping, badges, availability",
    "Updated discussion detail page with clickable provider labels and inline error retry UI",
    "Updated new discussion page with pre-selection of tenant defaults and model hints",
    "Updated admin page with new 'LLM Defaults' tab"
  ],
  "pains": [
    "Double-click issue with conditionally rendered ProviderPicker due to conflicting internal/external state management."
  ],
  "successes": [
    "Successfully implemented a robust, dynamic LLM selection system from schema to UI.",
    "Resolved ProviderPicker double-click issue with defaultOpen/onClose props, improving component reusability and UX.",
    "Clean typecheck and successful feature commit."
  ],
  "techStack": [
    "TypeScript",
    "tRPC",
    "Prisma",
    "Next.js",
    "React",
    "PostgreSQL",
    "Anthropic API",
    "OpenAI API",
    "Google API",
    "Kimi API"
  ]
}