Automating Documentation Quality with LLMs: Our Journey to the Docs Pipeline

Documentation is the lifeblood of any growing project, but let's be honest – it's often the first thing to fall by the wayside. Keeping docs up-to-date, consistent, and high-quality across multiple repositories and projects is a Herculean task. What if we could automate a significant chunk of that effort?

That's precisely the challenge we tackled this past session, culminating in the complete implementation of our new Docs Pipeline. It's an ambitious feature designed to bring sanity to documentation management, leveraging the power of Large Language Models (LLMs) to not just find problems, but to fix them.

A Bold Vision: The Automated Docs Pipeline

Our goal was clear: build an end-to-end system that could:

1. Scan repositories for all relevant documentation files.
2. Analyze their quality and relevance using an LLM.
3. Improve any docs falling below a defined threshold, again with LLM assistance.
4. Create GitHub Pull Requests with these improvements, ready for human review.

It's a multi-phase operation, and I'm thrilled to report that all six phases are now complete and committed. The codebase is type-check clean, lint clean, and boasts a perfect 180/180 test pass rate. That's a solid 33b4a7a milestone!

Under the Hood: Deconstructing the Pipeline

Building a feature of this scope required touching almost every layer of our stack. Here's a breakdown of the key components and architectural decisions:

1. Data Model & Persistence

To track pipeline runs and individual document items, we extended our Prisma schema:

• DocsRun: Represents a single execution of the pipeline, linked to a User, Tenant, Repository, and Project.
• DocsItem: Represents an individual documentation file processed within a DocsRun, including its status and improvement details.

// Excerpt from prisma/schema.prisma
model DocsRun {
  id           String      @id @default(uuid())
  createdAt    DateTime    @default(now())
  updatedAt    DateTime    @updatedAt
  status       DocsRunStatus
  phase        DocsPipelinePhase
  progress     Float
  // ... relations to User, Tenant, Repository, Project
  docsItems    DocsItem[]
}

model DocsItem {
  id           String      @id @default(uuid())
  createdAt    DateTime    @default(now())
  updatedAt    DateTime    @updatedAt
  filePath     String
  format       DocFormat
  category     DocCategory
  status       DocsItemStatus
  originalContent String?
  improvedContent String?
  prUrl        String?
  // ... relations to DocsRun
}

We also added robust Row-Level Security (RLS) policies for docs_runs and docs_items in prisma/rls.sql to ensure data isolation and security, a critical consideration for multi-tenant applications.

2. The Core Services: The Brains of the Operation

The heart of the Docs Pipeline lies in a suite of dedicated services, each responsible for a specific phase:

• src/server/services/docs/doc-scanner.ts: This AsyncGenerator is our repo explorer. It uses the GitHub API to traverse repository trees, identifying potential documentation files. It also intelligently classifies documents by DocCategory (e.g., README, API Reference, How-To Guide) based on heuristics.
• src/server/services/docs/relevance-scorer.ts: Before handing documents to the LLM, we apply heuristic scoring. memoryRelevance and qualityRelevance help us prioritize which documents need attention most, saving LLM tokens for the most impactful improvements.
• src/server/services/docs/doc-analyzer.ts: This is where the LLM magic begins. We batch documents (respecting a 50K character budget per batch) and send them to the LLM for a comprehensive quality analysis. The LLM identifies areas for improvement, inconsistencies, and clarity issues.
• src/server/services/docs/doc-improver.ts: Once analyzed, documents flagged for improvement enter this service. The LLM, enriched with relevant context (memory + consolidation), generates improved versions of the documentation, focusing on clarity, completeness, and adherence to best practices.
• src/server/services/docs/create-prs.ts: The final mile! This service takes the improved documents and programmatically creates GitHub Pull Requests, proposing the changes back to the original repository. Each PR is linked to its DocsItem.
• src/server/services/docs/pipeline.ts: The grand orchestrator. This main AsyncGenerator ties all the above services together, managing the flow from scanning to PR creation, ensuring smooth transitions between phases and robust error handling.

3. API & User Interface

To make this pipeline accessible and observable, we built:

• src/app/api/v1/events/docs-pipeline/[id]/route.ts: A Server-Sent Events (SSE) streaming endpoint. This allows our frontend to receive real-time updates on pipeline progress, showing which phase is active and the status of individual DocsItems.
• src/server/trpc/routers/docs-pipeline.ts: A robust tRPC router with 7 procedures: list, get, start, cancel, skipItem, retryPRs, and generateReport. This gives us full control over pipeline execution from the frontend.
• src/app/(dashboard)/dashboard/docs-pipeline/page.tsx: The main dashboard page for viewing a list of all DocsRuns and launching new ones via an intuitive dialog.
• src/app/(dashboard)/dashboard/docs-pipeline/[id]/page.tsx: A detailed view for a specific pipeline run, featuring the SSE-powered phase progress bar and a table displaying the status of each DocsItem.

We also added a "Docs Pipeline" navigation item to our sidebar, making it easily discoverable within the application.

Navigating the Bumps: Lessons Learned

No complex feature development is without its quirks. Here are a couple of "aha!" moments and critical fixes that emerged during this session:

Lesson 1: LLM API Consistency is Key

• The Problem: While implementing doc-analyzer.ts and doc-improver.ts, I instinctively reached for llm.chat(). However, TypeScript immediately flagged TS2339: Property 'chat' does not exist on type 'LLMProvider'.
• The Fix: A quick check of our LLMProvider interface revealed that the standard method for general LLM calls in our codebase is llm.complete().
• Takeaway: Even with strong typing, muscle memory can lead you astray. Always refer to the defined interfaces and contracts, especially when working with shared service layers. This ensures consistency and prevents runtime surprises.

Lesson 2: Extending Union Types for Dynamic Context

• The Problem: Our pipeline-context.ts module is responsible for loading summaries of previous pipeline runs. It had a predefined union type for pipelineType ("autofix" | "refactor"). When I tried to use "docs" as the new pipelineType, TypeScript rightfully complained: TS2345: Argument of type '"docs"' is not assignable to parameter of type '"autofix" | "refactor"'.
• The Fix: We extended the pipelineType union to include "docs" and added a new else-if branch in loadPreviousRunSummaries() to query the DocsRun model specifically.
• Takeaway: As your application grows and new features introduce new types of entities or processes, remember to revisit and extend existing type definitions, especially union types, to maintain type safety and ensure all code paths are correctly handled. This often goes hand-in-hand with updating data fetching logic.

What's Next on the Roadmap?

While the core pipeline is operational, our journey isn't over. Immediate next steps include:

1. Project-Level Docs Management: Extending the projects.docs sub-router with scan, create, update, delete, and enrich procedures to manage documentation directly within project contexts.
2. Enhanced Project Docs UI: Building out the UI for the project's Docs tab, allowing users to toggle all files, see relevance badges, perform CRUD actions, and trigger enrichment dialogs.
3. Unit Tests for relevance-scorer.ts: This pure function is an ideal candidate for focused unit tests to ensure its heuristics are always performing as expected.
4. End-to-End Test: A crucial E2E test to simulate starting a docs pipeline and verifying the SSE streaming through all phases, ensuring the entire flow works seamlessly.
5. RLS Policy Application: A reminder to apply the RLS policies to the production database: psql -f prisma/rls.sql.

This Docs Pipeline represents a significant leap forward in our mission to help developers maintain high-quality codebases. We're excited about the potential it unlocks for automated documentation, freeing up valuable developer time to focus on building new features.

{
  "thingsDone": [
    "Implemented DocsRun and DocsItem models with RLS",
    "Created core services: doc-scanner, relevance-scorer, doc-analyzer, doc-improver, create-prs",
    "Developed main AsyncGenerator orchestrator (pipeline.ts)",
    "Built SSE streaming endpoint for real-time updates",
    "Implemented tRPC router with 7 procedures for pipeline control",
    "Designed and implemented dashboard UI for pipeline runs and details",
    "Extended GitHub connector for file deletion",
    "Integrated docs context into report generation",
    "Updated pipeline context for 'docs' pipelineType",
    "Added 'Docs Pipeline' navigation item"
  ],
  "pains": [
    "Misuse of LLMProvider.chat() instead of LLMProvider.complete()",
    "Attempting to use new 'docs' pipelineType without extending existing union type in pipeline-context"
  ],
  "successes": [
    "All 6 phases of Docs Pipeline feature complete",
    "Codebase is type-check clean and lint clean",
    "180/180 tests passing",
    "Successfully integrated LLM-powered analysis and improvement",
    "Real-time progress updates via SSE",
    "Robust data model and security with RLS"
  ],
  "techStack": [
    "TypeScript",
    "Next.js",
    "React",
    "Prisma",
    "PostgreSQL",
    "tRPC",
    "LLMs (Large Language Models)",
    "GitHub API",
    "SSE (Server-Sent Events)",
    "AsyncGenerator"
  ]
}