← nyxcore-systems
7 min read

From Crawler Chaos to Mobile Nirvana: A Session in Bugs, Breakthroughs, and Beautiful Dashboards

Join me as I recount a challenging dev session, tackling a stubborn HTML stripping bug in our site crawler and rolling out a complete mobile-first CSS overhaul for our dashboard pages.

fullstackfrontendbackendweb-crawlermobile-firsttailwind-cssprismapostgresdevopslessons-learned

Every developer knows those sessions. The ones where you dive in with a clear plan, only to hit unexpected roadblocks, learn critical lessons the hard way, and emerge hours later, exhausted but exhilarated, with significant progress under your belt. This past Wednesday, March 11th, was one of those days.

My mission: First, squash a gnarly HTML stripping bug in our site crawler that was polluting our RAG (Retrieval Augmented Generation) data. Second, embark on a full mobile-first CSS overhaul for every single dashboard page. Ambitious? Absolutely. Achieved? You bet.

Taming the HTML Stripper: A Crawler's Tale

Our RAG system relies heavily on clean, well-formatted text. Imagine our dismay when we discovered that documents sourced from certain URLs were appearing in our system with their HTML tags intact, rather than properly stripped and parsed. This wasn't just a cosmetic issue; it was directly impacting the quality of our AI's responses.

The culprit turned out to be a two-pronged attack:

  1. The processDocument() Fallback: Our src/server/services/rag/document-processor.ts had a logic flaw. In processDocument(), if a local file wasn't found, it would always fall back to re-fetching from the sourceUrl without proper checks. The fix was simple but crucial: an fs.access() check to ensure the local file truly didn't exist before hitting the network again. This prevents unnecessary re-fetching and ensures we're processing the most accurate source.

    typescript
    // src/server/services/rag/document-processor.ts:397 (simplified)
async function processDocument(documentPath: string, sourceUrl?: string) {
  let content: string;
  try {
    // Check if local file exists and is accessible
    await fs.promises.access(documentPath, fs.constants.F_OK);
    content = await fs.promises.readFile(documentPath, 'utf8');
  } catch (error) {
    if (sourceUrl) {
      // Fallback to re-fetch only if local file truly not found
      content = await fetchAndProcess(sourceUrl);
    } else {
      throw new Error('Document content not found locally and no source URL provided.');
    }
  }
  // ... rest of processing logic
}

  2. The MimeType Mix-up: The second part of the puzzle was a subtle mimeType misconfiguration in our site-crawler-service.ts. It was incorrectly identifying text/html content as text/plain, which meant our HTML stripping logic wasn't even being triggered for certain documents. A quick change from text/html to text/plain (counter-intuitive, perhaps, but correct for how our pipeline expects raw text for stripping) resolved this.

With the bug squashed, the next step was a surgical cleanup. I deleted 153 tainted BetrVG documents from production to ensure a fresh start. This also necessitated manually creating the crawl_jobs table via SQL, as it was a new requirement not yet integrated into our Prisma migrations.

The result? Commit 0957b1b deployed, 318 tests passing, typecheck clean. The crawler now correctly processes documents, and our RAG system can breathe a sigh of relief.

The Mobile Makeover: Crafting a Responsive Dashboard

With the backend stabilized, it was time to shift gears to the frontend. Our dashboard, while functional on desktop, desperately needed a mobile-first overhaul. The goal wasn't just responsiveness, but a genuinely pleasant experience on smaller screens.

I started with a dedicated design doc (docs/plans/2026-03-11-mobile-first-css-design.md) and then an implementation plan (docs/plans/2026-03-11-mobile-first-css.md) outlining six key tasks. What made this particularly interesting was executing these tasks via "subagent-driven development"—essentially, breaking down the problem into small, well-defined prompts for an AI assistant, then meticulously reviewing and integrating its output.

Here's a breakdown of the key changes:

  1. Viewport & Safe Areas (4d43396):

    • Removed maximumScale:1 from the viewport meta tag for better zoom accessibility.
    • Added viewportFit:cover and safe-area padding to the mobile navigation.
    • Introduced a pb-20 on the main content area to prevent content from being hidden by the fixed mobile nav.

  2. Adaptive Sidebar Layout (7aee355):

    • The traditional desktop sidebar (hidden md:block) now transforms into a horizontal, scrollable tab bar on mobile (md:hidden).
    • The SidebarPageLayout component now dynamically adjusts its flex direction: flex-col on mobile, md:flex-row on desktop.

  3. Responsive Dialogs (1fc31a3):

    • Dialogs now take up more width on mobile (w-[calc(100%-2rem)]) while reverting to full width on larger screens (md:w-full).
    • Padding adjusted (p-4 md:p-6) and crucial max-h-[85vh] overflow-y-auto added to ensure dialog content is always scrollable and doesn't push off-screen.

  4. Smarter Tables (97861ee):

    • Our model usage tables, which had many columns (Cost, Calls, Duration, Energy), were overwhelming on mobile.
    • Key columns now use hidden md:table-cell, collapsing them on smaller screens.
    • A "tap-to-expand" functionality with a ChevronDown icon was added to reveal full row details on demand.

  5. Fluid Detail Pages (07817b5):

    • Pages like "Ipcha" and "Projects detail" often featured horizontally laid-out elements that broke on mobile.
    • I applied flex-col gap-2 sm:flex-row to five specific layout elements, ensuring they stack vertically on smaller screens and revert to horizontal on small-to-medium viewports.

  6. Mobile Navigation Polish (19ea037):

    • "Style" in the mobile nav was replaced with "Memory" (represented by a Brain icon) to better reflect its function.
    • A critical fix for the sidebar component within a mobile sheet: the sidebar had hidden md:flex, making it invisible within the sheet below the md: breakpoint. The workaround involved adding a className prop to Sidebar and passing className="flex w-full" from the Sheet component to override the default hidden behavior.

Lessons Learned the Hard Way (The "Pain Log")

No session is complete without a few head-scratching moments. These often turn into the most valuable lessons:

  • SSH Heredocs vs. docker exec psql:

    • Tried: Piping multi-statement SQL via SSH heredoc to docker exec psql.
    • Failed: The heredoc syntax didn't pass through the SSH and Docker exec chain correctly, leading to syntax errors.
    • Lesson: For complex SQL on a remote Docker container, either create a .sql file and copy it over, or (for simpler cases) use the single-line -c flag with carefully escaped quotes. It's tedious, but reliable.

  • npx prisma db push on Production:

    • Tried: Running npx prisma@5.22.0 db push inside the production container to apply a new table.
    • Failed: db push is designed for rapid prototyping and development environments. It can irrevocably drop existing data, specifically pgvector embedding columns in our case, if not handled with extreme care.
    • Lesson: NEVER use prisma db push on production. Always rely on prisma migrate deploy for applying migrations, or for ad-hoc schema changes, manual CREATE TABLE / ALTER TABLE statements via docker exec psql. This was a near-disaster averted by a quick rollback.

  • Responsive Component Visibility within Other Components:

    • Tried: Placing a Sidebar component (which had hidden md:flex) directly inside a Sheet component for mobile navigation.
    • Failed: Below the md: breakpoint, the Sidebar's hidden class took precedence, making it invisible inside the mobile sheet.
    • Lesson: When nesting responsive components, be acutely aware of how parent/child display and visibility utility classes interact. Design components with override props (like className) to allow parents to dictate their children's visual state in specific contexts.

Looking Ahead

While the immediate goals are complete, the journey continues:

  1. Verification: The user needs to re-crawl the BetrVG documents and verify clean text extraction. I also need to test the mobile UI on a real device to ensure the horizontal tab bar, dialogs, table expand, and sidebar sheet behave as expected.
  2. Security: The crawl_jobs table is currently unprotected. An RLS (Row Level Security) policy needs to be considered and implemented.
  3. Code Cleanup: Consolidating the duplicate .no-scrollbar and .scrollbar-none utilities in globals.css is a minor but important cleanup task.
  4. Polish: A follow-up "touch target" polish pass will ensure all interactive elements meet the 44px minimum recommendation for mobile accessibility.

This session was a microcosm of full-stack development: debugging a backend data pipeline, meticulously crafting a frontend user experience, and navigating the operational pitfalls of deployment. It's a reminder that every line of code, every design decision, and every painful lesson learned contributes to a more robust and user-friendly product.

json
{
  "thingsDone": [
    "Fixed HTML stripping bug in document processor",
    "Corrected crawler mimeType handling",
    "Deleted tainted production documents",
    "Manually created crawl_jobs table on production",
    "Designed mobile-first CSS approach",
    "Implemented 6 tasks for mobile-first CSS overhaul",
    "Deployed all changes to production"
  ],
  "pains": [
    "SSH heredoc failure for multi-statement SQL",
    "Accidental `prisma db push` on production dropping columns",
    "Responsive component visibility conflict within nested components"
  ],
  "successes": [
    "Successful bug fix for data integrity",
    "Complete mobile-first redesign of dashboard UI",
    "Successful deployment with all tests passing",
    "Effective use of subagent-driven development for frontend tasks",
    "Learning critical lessons about production database management"
  ],
  "techStack": [
    "TypeScript",
    "Node.js",
    "Tailwind CSS",
    "React",
    "Next.js",
    "Prisma",
    "PostgreSQL",
    "Docker",
    "SSH"
  ]
}