Bringing Project Docs to Life: GitHub Integration, Dynamic Markdown, and a Dash of AI Magic

Documentation is often the unsung hero of a thriving project, yet it's frequently scattered, outdated, or hard to find. We've all been there: digging through a repo for a README.md that's half-finished, or hunting for a design doc buried in an old Notion page. Our goal? To make documentation a first-class citizen, seamlessly integrated into our project dashboard.

This past sprint, we tackled exactly that, rolling out a shiny new "Docs" tab on our project detail pages. This isn't just a static link; it's a fully interactive experience that fetches, renders, and enhances our project's knowledge base, all powered by our linked GitHub repositories.

Let's dive into how we built it, what challenges we faced, and the cool features we brought to life.

The Vision: Docs, Integrated and Intelligent

Imagine clicking on a project in your dashboard and instantly having access to its core documentation, organized, searchable, and beautifully rendered. That was the dream. Our new "Docs" tab offers a three-state experience:

Docs List: A grid view of all .md files in the project's /docs/ folder.
Summary Card: A quick peek at a document's title and excerpt.
Full Rendered Doc: The complete markdown content, brought to life.

The Backend: GitHub as Our Source of Truth

The first step was connecting to GitHub. We leveraged trpc to create a docs sub-router within our existing projects router.

docs.list: This endpoint uses a custom fetchRepoTree utility to recursively list all markdown files within the /docs/ directory of the linked GitHub repository.
docs.get: Given a file path, this endpoint uses fetchFileContent to retrieve the raw markdown content. It also intelligently extracts the title and a short summary from the document's frontmatter or initial paragraphs.

This approach ensures that our documentation always reflects the latest state of the repository, making GitHub the single source of truth for project knowledge.

typescript

// src/server/trpc/routers/projects.ts (simplified)
import { createTRPCRouter, publicProcedure } from '../trpc';
import { z } from 'zod';
import { fetchRepoTree, fetchFileContent } from '../utils/github';

export const projectsRouter = createTRPCRouter({
  // ... existing project routes ...
  docs: createTRPCRouter({
    list: publicProcedure
      .input(z.object({ githubOwner: z.string(), githubRepo: z.string() }))
      .query(async ({ input }) => {
        return fetchRepoTree(input.githubOwner, input.githubRepo, 'docs');
      }),
    get: publicProcedure
      .input(z.object({ githubOwner: z.string(), githubRepo: z.string(), path: z.string() }))
      .query(async ({ input }) => {
        const content = await fetchFileContent(input.githubOwner, input.githubRepo, input.path);
        // Basic title/summary extraction logic here
        const title = content.match(/^#\s(.+?)(?:\n|$)/)?.[1] || 'Untitled';
        const summary = content.split('\n\n')[1]?.substring(0, 150) + '...' || 'No summary available.';
        return { content, title, summary };
      }),
  }),
});

The Frontend: A Rich Rendering Experience

On the client side, our src/app/(dashboard)/dashboard/projects/[id]/page.tsx now sports a new DocsTab component. This component orchestrates the fetching and display of documentation, transitioning smoothly between the list, summary, and full-doc views. We also added a neat BookOpen icon to the tab trigger for a touch of visual flair.

But the real magic happens in our src/components/markdown-renderer.tsx. We supercharged our markdown renderer to support:

1. Mermaid Diagrams

Technical documentation often benefits from diagrams. We integrated Mermaid to allow developers to embed flowcharts, sequence diagrams, and more directly within their markdown. We used a dynamic import for MermaidDiagram to keep our initial bundle size lean and ensured it renders SVG output with a dark theme compatible style.

markdown

```mermaid
graph TD;
    A[Start] --> B{Decision};
    B -- Yes --> C[Action 1];
    B -- No --> D[Action 2];
    C --> E[End];
    D --> E;


#### 2. LaTeX-style Math Equations

For projects involving algorithms, data science, or complex logic, math is crucial. We added `remark-math` and `rehype-katex` plugins to support both inline (`$E=mc^2$`) and block (`$$ \sum_{i=1}^n i = \frac{n(n+1)}{2} $$`) equations, rendering them beautifully with KaTeX.

#### 3. Smart Internal Repo Links

A subtle but powerful feature is our new `RepoLink` component. Often, documentation refers to specific source code files (e.g., "See `src/server/utils/auth.ts` for details"). Our `RepoLink` component automatically converts these file paths (matching patterns like `src/`, `lib/`, `app/`) into clickable GitHub blob URLs, making it incredibly easy to jump directly to the referenced code.

These enhancements required a few new dependencies in `package.json`: `mermaid`, `remark-math`, `rehype-katex`, and `katex`.

## The Great Documentation Migration (with a little AI help)

Beyond the technical implementation, a significant part of this effort involved migrating our existing, often monolithic, documentation. We split large `/tmp/nyxcore-doc-part{1,2,3}.md` files into individual, focused sections (e.g., `01-executive-summary.md` through `11-real-world-examples.md`).

Here's where things got interesting: we leveraged background agents to help write and structure new documentation sections. Sections 12 (auto-fix), 13 (refactor), 17 (github-connector), 18 (analytics-dashboard), and 19 (injection-diagnostics) were all drafted and committed by these agents. A few more — 14 (code-analysis), 15 (action-points), 16 (discussion-service), and 20 (sidebar-active-processes) — are still being finalized as I write this. This blend of human curation and AI-assisted generation is proving to be a powerful workflow.

## Lessons from the Trenches: Overcoming Development Hurdles

No development sprint is without its snags. Here are a few "aha!" moments and workarounds we discovered:

*   **The Root-Owned npm Cache:**
    *   **Problem:** Encountered `EACCES` errors due to a root-owned npm cache in `~/.npm/_cacache/`, preventing `npm install` from running.
    *   **Lesson/Workaround:** Instead of fighting permissions or trying to `sudo` everything, a quick and reliable fix was to tell npm to use a temporary cache directory: `npm install --cache /tmp/npm-cache-nyxcore`. This allowed us to proceed without getting bogged down in system-level permissions, a good reminder that sometimes a temporary workaround is better than a deep dive into an unrelated problem.

*   **TypeScript Regex Flag Compatibility:**
    *   **Problem:** Attempted to use the `/s` (dotall) regex flag in TypeScript (`/^(.+?)(?:\n\n|\n#|$)/s`) for easier multi-line matching, only to hit `TS1501: This regular expression flag is only available when targeting 'es2018' or later`. Our current target was older.
    *   **Lesson/Workaround:** The classic fallback: replace `.` with `[\s\S]`. So, our regex became `/^([\s\S]+?)(?:\n\n|\n#|$)/`. Always be mindful of your project's target environment and language feature compatibility!

*   **"Prompt is Too Long" - Agent Edition:**
    *   **Problem:** When tasking a single background agent with merging a large number of source files into a single documentation section, we ran into "Prompt is too long" errors (a common limitation with large language models).
    *   **Lesson/Workaround:** Break down the problem! We split the task into multiple parallel agents with narrower scopes (e.g., one agent for sections 12-16, another for 17-20), and even used a separate Bash agent for the initial file-splitting operation. This not only avoided prompt length issues but also improved processing efficiency.

## What's Next?

As those last few documentation sections finish writing themselves, our immediate next steps are:

1.  Commit and push the remaining generated documentation files.
2.  Thoroughly verify the "Docs" tab end-to-end on a project with a linked GitHub repo.
3.  Confirm Mermaid diagrams render correctly in dark theme and as SVG.
4.  Consider adding a `docs/00-index.md` README file to serve as a comprehensive table of contents for the entire `/docs/` folder.

This has been a hugely satisfying sprint, bringing our project documentation out of the shadows and into the core of our application. By integrating GitHub, leveraging rich markdown, and even experimenting with AI-assisted content generation, we're making it easier than ever for developers to find, consume, and contribute to project knowledge.

---

```json
{
  "thingsDone": [
    "Implemented Docs tab on project detail page",
    "Integrated GitHub for fetching documentation files",
    "Added Mermaid diagram support to markdown renderer",
    "Added LaTeX-style math support (KaTeX) to markdown renderer",
    "Developed RepoLink component for smart internal GitHub code links",
    "Migrated monolithic documentation into individual files in a /docs/ folder",
    "Utilized background agents to generate new documentation sections"
  ],
  "pains": [
    "npm cache EACCES permissions error",
    "TypeScript regex /s flag compatibility issue (target es2018+)",
    "Background agent 'prompt too long' error for large doc merge tasks"
  ],
  "successes": [
    "Reliable npm install with temp cache workaround",
    "Successful regex adaptation using [\\s\\S]",
    "Effective task parallelization for background agents",
    "Seamless integration of rich markdown features (Mermaid, Math, RepoLink)",
    "Automated documentation generation and migration workflow"
  ],
  "techStack": [
    "TypeScript",
    "React",
    "Next.js",
    "tRPC",
    "GitHub API",
    "Markdown",
    "remark-math",
    "rehype-katex",
    "katex",
    "mermaid",
    "Background Agents (AI/scripting)"
  ]
}