NyxBook Just Got Smarter: Seamless GitHub & Zip Imports, Plus a Sleek New Sidebar!

Hey fellow developers and digital storytellers! Today, I'm thrilled to pull back the curtain on a significant update for NyxBook – our beloved tool for managing complex narrative projects. The goal for this session was clear: make it radically easier to get your existing narrative content into NyxBook, and while we were at it, give our main application sidebar a much-needed facelift.

Consider it done. All features are implemented, type-checked, lint-clean, and fresh off the code review press. Let's dive into what's new!

The Quest for Seamless Content Ingestion

One of the biggest hurdles when adopting a new tool is migrating existing data. For NyxBook users, this often meant manually copying and pasting chapters, character profiles, and beat sheets. No more! We've introduced two powerful new ways to import your content: directly from GitHub repositories and via .zip file uploads.

GitHub Integration: Your Repo, Your Book

Imagine pointing NyxBook at your GitHub repository, and with a few clicks, your entire book structure, characters, and beats are automatically imported. That's precisely what we've built.

• How it Works:
  • Smart Scanning: Our new checkGitHubForBook() service intelligently scans your chosen GitHub repository for a book/ directory – our standard convention for structured narrative content.
  • API-Powered Fetching: Once found, importFromGitHub() leverages the GitHub API to fetch your content. It also leverages our existing BYOK (Bring Your Own Key) pattern for secure GitHub token resolution.
  • Intelligent Parsing: Using our robust, shared parsing services (more on those in a moment), we extract characters, beats, chapters, and personas, creating rich database records for your project.
  • Frontend Magic: On the dashboard, a new, searchable combobox lets you quickly find your repositories. Select one, and NyxBook will auto-detect the book/ directory structure, giving you immediate feedback.
• Under the Hood: We've added checkGitHub and importGitHub procedures to our tRPC router, complete with rigorous regex validation for GitHub owner (max 39 chars) and repo names (max 100 chars) to keep things secure. Branch names are also capped at 256 characters.

Zip Uploads: Drag, Drop, and Done

For those who prefer to keep their work local or simply have a structured .zip archive, we've introduced a powerful drag-and-drop zip upload feature.

• Simple & Secure: Just drag your .zip file onto the designated zone on the dashboard.
• Smart Extraction: Our backend handles the multipart/form-data upload, extracts the contents using JSZip, and intelligently searches for your book/ root directory with findBookRoot().
• Safety First: We've implemented strict limits (20MB / 200 entries) and .md-only file filtering, coupled with robust path traversal protection, ensuring only your content gets in, and nothing malicious. Temporary directories are, of course, cleaned up swiftly.

The Power of Shared Parsers

A core part of making these imports robust and consistent was refactoring our parsing logic. We've consolidated and exported key parser functions like parseCharacters, parseBeats, splitDualLayer, and countWords into a shared service. This ensures that whether your content comes from GitHub, a zip file, or even a server path (though we're revisiting that one), it's processed identically and efficiently.

Crucially, we've also sanitized metadata, changing sourcePath to a generic source: "filesystem" to ensure no sensitive filesystem paths leak into your records – a key security enhancement from our code review.

A Fresh Look: The Sidebar Redesign

Beyond the powerful new import capabilities, we also took the opportunity to give NyxBook's main application sidebar a significant aesthetic and functional overhaul.

Inspired by the clean, focused InPageSidebar style that users loved, we've brought that same polish to the global navigation. Expect:

• Compact & Clear: text-xs items and text-[9px] group labels with reduced opacity (/60) for better hierarchy.
• Modern Aesthetics: rounded links, a subtle bg-nyx-accent/10 active state, and refined gap-2 spacing create a fresh, modern feel.
• Functional Elegance: Icons are now h-3.5, and the sidebar itself intelligently adjusts between w-12 (collapsed) and w-48 (expanded) widths, offering a more streamlined and intuitive navigation experience.

The result is a sidebar that feels less cluttered, more inviting, and perfectly aligned with the clean design language of NyxBook.

The Developer's Journey: Acknowledging the "Oops"

No development session is complete without its little quirks and learning moments. For this one, it was a classic TypeScript/JavaScript iteration gotcha.

During the development of findBookRoot() for the zip upload, I instinctively reached for a for...of loop on a Set<string> to iterate through directories. TypeScript, however, quickly reminded me of TS2802: Set can only be iterated with '--downlevelIteration'. This compiler flag isn't enabled in our tsconfig (and for good reason, to prevent unexpected runtime behavior in older environments). The fix was simple: Array.from(dirs) before iteration, but it's a good reminder of project-specific quirks, even those documented in our CLAUDE.md knowledge base!

On a more serious note, our internal code review flagged an existing (pre-dating this session) server-path import procedure as a potential path traversal risk – it accepts any filesystem path. While not directly part of this session's scope, it's now on the immediate roadmap for restriction to /tmp/nyxcore-uploads/ or complete removal, especially with the new, safer GitHub and zip import options. Security is an ongoing commitment!

What's Next?

With these features now polished and ready, the immediate next steps involve thorough end-to-end testing of both GitHub and zip imports. We're also looking at wrapping our database writes in prisma.$transaction() for atomicity (as flagged in review item #6) and considering further component extraction to keep our dashboard page lean (~530 lines, manageable but could split).

This update significantly enhances how you can bring your creative worlds into NyxBook, making it an even more powerful companion for your narrative projects. We can't wait to see what you'll create!

{
  "thingsDone": [
    "Implemented GitHub repository import service (scanning for book/, fetching content via API, parsing, creating DB records for Book, Characters, Beats, Chapters, Personas)",
    "Developed REST POST endpoint for .zip file uploads (multipart/form-data, JSZip extraction, findBookRoot() auto-detection, path traversal protection, 20MB/200-entry limits, .md-only filter, temp dir cleanup)",
    "Redesigned the main application sidebar to match InPageSidebar style (text-xs items, rounded links, bg-nyx-accent/10 active state, text-[9px] group labels with /60 opacity, gap-2 spacing, w-12/w-48 widths, h-3.5 icons)",
    "Refactored and exported shared parsing functions and types (parseCharacters, parseBeats, splitDualLayer, countWords, CharacterProfile, BeatEntry, DualLayerContent, ImportResult) for reusability across import methods",
    "Applied security fixes: robust input validation for GitHub owner/repo (regex, max lengths), metadata sanitization (sourcePath removed, changed to source: 'filesystem')",
    "Integrated GitHub import with tRPC (checkGitHub query, importGitHub mutation) and frontend UI (searchable combobox, auto-scanning)",
    "Addressed TypeScript Set iteration compatibility issue by using Array.from() for broader browser support"
  ],
  "pains": [
    "Encountered TypeScript TS2802 error when iterating Set<string> without '--downlevelIteration' flag, requiring Array.from() workaround.",
    "Identified and noted pre-existing path traversal risk in the server-path import procedure (accepts any filesystem path), flagged for future restriction or removal.",
    "Ensuring robust and secure handling of multipart/form-data and zip file contents, including path traversal protection and file type/size limits, while maintaining user-friendliness."
  ],
  "successes": [
    "Successfully integrated two major new content import methods (GitHub, Zip) providing flexible and efficient ways to ingest narrative projects.",
    "Achieved a significant UI/UX improvement with the main sidebar redesign, enhancing navigation and aligning with user-preferred aesthetics.",
    "Created a modular and reusable set of parsing services, improving code maintainability, consistency, and reducing duplication.",
    "Passed typecheck and linting, with all code review fixes applied, ensuring high code quality and security standards.",
    "Enhanced application security through rigorous input validation, metadata sanitization, and responsible handling of file uploads.",
    "Successfully integrated JSZip for client-side zip file processing and GitHub API for repository interactions."
  ],
  "techStack": [
    "TypeScript",
    "Next.js",
    "tRPC",
    "Prisma",
    "JSZip",
    "Tailwind CSS",
    "GitHub API",
    "PostgreSQL (via Prisma)",
    "React (for frontend components)"
  ]
}