← nyxcore-systems
6 min read

Engineering a Knowledge Hub: Integrating Dynamic Docs, Mermaid, and Math

A deep dive into integrating dynamic documentation features like Mermaid diagrams and LaTeX math into our projects dashboard, alongside a major content migration and key lessons learned from common development hurdles.

Next.jsTypeScriptMarkdownMermaid.jsKaTeXGitHub APIDocumentationFrontendBackendDevOpsLLMAI

Every great project needs great documentation. But what makes documentation truly great? It's not just about content; it's about accessibility, clarity, and the ability to convey complex ideas effortlessly. Recently, our mission was to elevate our project documentation experience, turning a static collection of files into a dynamic, interactive knowledge hub directly within our dashboard.

This session was all about bringing our internal intelligence documentation to life, integrating powerful rendering capabilities, and streamlining content management.

The Core Mission: A Three-Pronged Attack

Our primary goal for this development session was ambitious yet clear:

  1. Implement a dedicated "Projects Docs" tab: A central place for all project-related documentation.
  2. Unleash the power of Mermaid and Math: Enable rendering of Mermaid diagrams and LaTeX math formulas directly within our Markdown.
  3. Migrate and structure our intelligence docs: Move 20 critical documentation sections into a clean, GitHub-backed docs/ folder, each as an individual Markdown file.

By the end of the session, all targets were hit. The dev server purred on port 3000, and our documentation was ready for its grand debut.

Diving Deep: The Implementation Journey

Let's break down the key architectural and code changes that made this possible.

Backend: Powering Docs with GitHub Integration

Our backend, built with tRPC, needed to become the bridge between our application and our GitHub repositories.

In src/server/trpc/routers/projects.ts:

  • We integrated fetchRepoTree from our github-connector to efficiently list files.
  • A new docs sub-router was introduced, housing two crucial procedures:
    • docs.list({ projectId }): This procedure now fetches the entire GitHub repository tree for a given project, filtering specifically for .md files within the docs/ directory. It returns a concise list of { path, name } pairs, ready for display.
    • docs.get({ projectId, filePath }): Given a project and a specific file path, this procedure retrieves the raw content of the Markdown file. It intelligently extracts the primary title (from the first # heading) and a summary (the first paragraph) for quick previews, using a robust [\s\S]+? regex to handle multiline content.

Frontend: A Smarter Markdown Renderer

The true magic happens in src/components/markdown-renderer.tsx, our centralized Markdown processing component. This file received a significant upgrade:

  • Math Rendering: We integrated remark-math and rehype-katex to transform $inline$ and $$block$$ LaTeX syntax into beautiful, rendered math equations. This is critical for explaining complex algorithms and formulas.
  • Mermaid Diagrams: To visualize workflows and architectures, we built a dedicated MermaidDiagram component. It uses dynamic imports for mermaid to keep our bundle size lean, applies a dark theme for consistency, and renders Mermaid code blocks as SVG via dangerouslySetInnerHTML.
  • Intelligent Code Blocks: Our CodeBlockWrapper now inspects the language specified for code blocks. If it detects language-mermaid, it gracefully delegates rendering to our new MermaidDiagram component.
  • GitHub Link Rewriting: A clever RepoLink component was added to automatically rewrite internal source code references (e.g., src/server/ or docs/) into proper GitHub blob URLs. This means links within our Markdown now point directly to the relevant files in the GitHub repo, improving navigability for developers.
  • Flexibility & Performance: We added optional githubOwner and githubRepo props to MarkdownRendererProps to enable the link rewriting, and judiciously used useMemo for the components object to prevent unnecessary re-renders.

UI: The "Projects Docs" Tab Comes Alive

The user interface for our new documentation hub was integrated directly into the project dashboard:

  • In src/app/(dashboard)/dashboard/projects/[id]/page.tsx:
    • We added the BookOpen icon from lucide-react for a visually appealing "Docs" tab.
    • The new DocsTab component was developed, managing three distinct states: a grid view of all available documents, a summary card view when a document is selected, and the full document view for reading.
    • A new <TabsTrigger value="docs"> and <TabsContent value="docs"> were slotted into the dashboard after the existing "Actions" tab, making the docs easily accessible.
    • The DocsTab component orchestrates data fetching using our new tRPC procedures: trpc.projects.docs.list for the document list and trpc.projects.docs.get for individual document content.

Dependencies

To support these new features, we introduced a few key packages:

  • mermaid: For diagramming.
  • remark-math, rehype-katex, katex: For LaTeX math rendering.

The Great Documentation Migration

Beyond the technical features, a significant part of this session involved content management. We successfully migrated all 20 sections of our core intelligence documentation into the new docs/ structure. These sections, previously scattered or in monolithic files, are now individual, discoverable Markdown files:

  • Foundation: Sections 01-11 were meticulously split from larger /tmp/nyxcore-doc-part{1,2,3}.md files.
  • Core Systems: New sections like 12: Auto-Fix Pipeline, 13: Refactor Pipeline, 14: Code Analysis, 15: Action Points, and 16: Discussion Service detail the sophisticated inner workings of our platform, from multi-phase pipelines to LLM-driven analysis and collaborative discussion features.
  • Infrastructure & Analytics: Sections 17: GitHub Connector, 18: Analytics Dashboard, 19: Injection Diagnostics, and 20: Sidebar & Active Processes cover critical components like our BYOK GitHub integration, comprehensive analytics, context measurement, and UI navigation.

This structured approach makes our documentation not only accessible but also maintainable and extensible.

Lessons Learned: Navigating Development Hurdles

No development sprint is without its challenges. Here are a few "gotchas" and our effective workarounds:

  • NPM Cache Permissions (EACCES):

    • Problem: Running npm install after a npm cache clean --force would consistently fail with EACCES errors due to root-owned files in the ~/.npm/_cacache/ directory. Attempting sudo chown in a non-interactive shell is a risky proposition.
    • Solution: We adopted a reliable workaround: npm install --cache /tmp/npm-cache-nyxcore. This directs npm to use a temporary, user-owned cache directory, avoiding system-wide permission conflicts altogether. It's safe to delete after the install.

  • TypeScript and the /s Regex Flag:

    • Problem: We initially tried to use the /s (dotAll) regex flag in TypeScript (e.g., /.+?/s) for multiline matching when extracting document summaries. This resulted in a TS1501 error, indicating that the /s flag requires --target es2018 or later, which wasn't our current configuration.
    • Solution: The classic workaround came to the rescue: replacing . with [\s\S]. Our regex for extracting the first paragraph became /^([\s\S]+?)(?:\n\n|\n#|$)/, effectively matching any character, including newlines, without needing the /s flag.

  • LLM Context Window Limitations:

    • Problem: When attempting to use a single large AI agent to merge three existing documentation parts and write nine new sections, we hit the dreaded "Prompt is too long" error. The sheer volume of source files exhausted the LLM's context window.
    • Solution: We pivoted to a more modular approach: parallelizing agents with narrower scopes. We used separate agents for sections 12-16 versus 17-20 and even employed a dedicated Bash agent for the initial file splitting. This distributed the cognitive load, allowing the LLMs to operate within their context limits effectively.

What's Next?

With the core implementation complete, our immediate next steps involve thorough verification and minor enhancements:

  1. End-to-end Verification: Confirm the Docs tab functions perfectly on a live project with linked GitHub repos containing /docs/*.md files.
  2. Mermaid Theme Check: Ensure Mermaid SVG renders correctly in dark theme, looking out for any Flash of Unstyled Content (FOUC) or double-initialization issues.
  3. Index Document: Consider adding a docs/00-index.md file to serve as a README, listing all 20 sections with internal links for better navigation.
  4. Existing Callsites: Verify that existing MarkdownRenderer callsites (those not passing githubOwner/githubRepo) continue to function correctly, as these props are optional.

This session marked a significant leap forward in our project's documentation capabilities. By integrating dynamic rendering, smart content management, and learning from our challenges, we've built a robust and engaging knowledge hub that will serve our team well into the future.