← nyxcore-systems
9 min read

From Crawler Quirks to Mobile Magic: Dissecting Our Latest Production Push

Join us as we recount a recent production deployment, tackling elusive crawler bugs, overhauling our mobile UI with a dedicated CSS pass, and refining the user experience from the ground up.

web-developmentfrontendbackendcssmobile-firstbug-fixingdevopstailwind-cssproduction-deployment

Every once in a while, a development session feels like a mini-saga. You start with a clear goal, encounter unexpected twists, wrestle with stubborn issues, and emerge victorious, leaving a trail of commits and lessons learned. This past week culminated in one such session, where we pushed a significant update to production, addressing everything from arcane crawler bugs to a full mobile-first CSS overhaul.

The mission was clear: iron out some critical HTML processing bugs, give our mobile experience the dedicated attention it deserved, and polish the navigation to a shine. The good news? All complete and deployed. Now, for the real fun: user testing on a real device to catch those last elusive quirks.

Let's break down what went into this push, the challenges we faced, and the solutions we implemented.

Taming the Crawler: When HTML Goes Astray

Our system relies heavily on a robust crawler to ingest and process documents. Recently, we noticed some anomalous behavior: certain documents were coming through with stripped HTML, leading to incomplete or malformed content. This wasn't just an aesthetic issue; it impacted the core functionality of our document processing.

The investigation led us down two paths:

  1. The HTML Stripping Bug: Digging into document-processor.ts, specifically line 397, we uncovered a subtle bug. Our processor was sometimes re-fetching content from the sourceUrl even when a local file was already available. This re-fetch, under specific conditions, was inadvertently triggering a more aggressive HTML stripping logic. The fix was straightforward but critical: prioritize the local file over a redundant sourceUrl re-fetch. A single commit, 0957b1b, secured this.

  2. The MIME Type Mix-up: Simultaneously, we found a related issue in site-crawler-service.ts:333. Our crawler was incorrectly assigning text/plain to certain HTML documents instead of text/html. This misclassification meant our HTML processing pipeline wasn't being correctly applied, contributing to the content degradation. Correcting the mimeType assignment ensures that HTML content is always treated as such, allowing our parsers to do their job properly.

The fallout from these bugs meant we had 153 "tainted" BetrVG (German Works Constitution Act) documents in our production database. These needed to go. After a careful deletion, we also spun up a new crawl_jobs table on production via manual SQL (a necessary evil when bypassing Prisma migrations for immediate data-layer changes). This paves the way for a clean re-crawl and verification.

Embracing Mobile-First: A CSS Overhaul Journey

This was the lion's share of the work, driven by a dedicated "subagent" (our internal term for a focused, task-oriented development sprint). The goal was to transform our application into a truly mobile-first experience, not just a scaled-down desktop version.

Here’s how we tackled it, commit by commit:

1. The Viewport Foundation (4d43396)

The very first step for any modern mobile web app is getting the viewport right. We made two key adjustments:

  • Removed maximumScale:1: This is an accessibility win. Forcing maximumScale:1 prevents users from pinching to zoom, which can be critical for those with visual impairments. We believe in empowering the user.
  • Added viewportFit:cover and Safe-Area Padding: This ensures our content gracefully handles device notches and rounded corners, utilizing the full screen real estate without content spilling into unusable areas.

2. Sidebar Transformation: From Vertical to Horizontal Tabs (7aee355)

On desktop, a persistent sidebar is great. On mobile, it's a screen hog. We reimagined the in-page sidebar, converting it into a horizontal, scrollable tab bar specifically for mobile viewports. This was achieved using responsive utility classes like md:hidden (hide on medium screens and up) and hidden md:block (hide on small screens, show on medium and up) to conditionally render different layouts. This frees up precious vertical space.

3. Dialogs Done Right for Mobile (1fc31a3)

Modal dialogs often look cramped or oversized on mobile. We refined our dialog component:

  • Mobile Margins: w-[calc(100%-2rem)] ensures the dialog takes up almost the full width, with a comfortable 1rem margin on each side.
  • Padding Adjustments: p-4 md:p-6 provides appropriate internal padding that scales with the screen size.
  • Scrollable Content: max-h-[85vh] overflow-y-auto ensures that if a dialog's content exceeds the screen height, it becomes scrollable rather than overflowing, maintaining usability.

4. Model Usage Table: Smart Responsiveness (97861ee)

Complex data tables are notorious for breaking on mobile. For our model usage table, we opted for a progressive disclosure approach:

  • Conditional Columns: Four less critical columns now use hidden md:table-cell, meaning they are hidden on mobile but appear on larger screens.
  • Tap-to-Expand Rows: On mobile, users can tap a row to expand it and reveal the hidden details. This keeps the initial view clean while still providing full data access.

5. Layout Element Refinements (07817b5)

Across various detail pages (Ipcha, Projects), we adjusted layout elements to stack vertically on small screens and horizontally on larger ones using flex-col gap-2 sm:flex-row. This is a classic responsive pattern that ensures readability and optimal use of space.

Navigating the Nuances: Mobile Sidebar & UI Polish

Beyond the core CSS overhaul, we dedicated time to refining the mobile navigation and sidebar experience.

  • "Style" to "Memory" with Brain Icon (19ea037): A small but significant UX change. The "Style" navigation item, which leads to our memory management features, was renamed to "Memory" and given a distinct brain icon. This makes its purpose much clearer and more intuitive for users.
  • Sidebar Sheet Fix (19ea037): This one was a bit of a head-scratcher, leading to a valuable lesson (more on that below). We reuse our <Sidebar> component inside a <Sheet> component for mobile. The issue was that the hidden class applied to the sidebar for desktop contexts was making it invisible even inside the sheet on mobile. The fix involved adding a className prop to the Sidebar component, allowing the Sheet to pass className="flex w-full" to override the hidden property when rendered in a mobile sheet context.
  • Sidebar Mobile Polish (bcdb179): Further mobile refinements included hiding the collapse toggle and heartbeat animation via hidden md:flex on small screens. The logo, however, remains always visible, maintaining brand presence.

Lessons from the Trenches (The "Pain Log" Transformed)

Not everything goes smoothly. Here are some critical lessons learned from the "pain" points of this session:

1. Component Reusability vs. Contextual Styling

The Problem: We tried to reuse our <Sidebar> component directly within a <Sheet> for mobile navigation. The Pitfall: Our <Sidebar> component had hidden md:flex applied at its root, intending to hide it on mobile unless it was part of the main desktop layout. This caused it to remain hidden even when placed inside the <Sheet> component, which was supposed to make it visible on mobile. The Lesson: When reusing components, especially UI components with strong responsive styling, be acutely aware of their default states and how they interact with the context they're placed in. Explicitly overriding styles for specific contexts (like passing a className prop to force flex w-full) is often necessary to prevent unexpected hidden elements. Design components to be as context-agnostic as possible, or provide clear prop-based escape hatches for contextual overrides.

typescript
// Inside the Sheet component (simplified)
<Sheet>
  <SheetContent>
    {/* The fix: Pass a className to override default hidden state */}
    <Sidebar className="flex w-full" />
  </SheetContent>
</Sheet>

2. Shell Scripting Gotchas: SSH & Heredocs

The Problem: We needed to execute a multi-line SQL script on a production PostgreSQL instance running inside Docker, accessed via SSH. Our initial attempt involved using an SSH heredoc with docker exec psql. The Pitfall: Heredocs, while powerful for multi-line input, don't always play nicely when piped through ssh and then into docker exec. The shell on the remote machine might interpret the heredoc delimiters differently, or the docker exec command might not receive the input as expected. The Lesson: For simple, single-line commands executed via ssh and docker exec, it's often safer and more reliable to use the -c flag with escaped quotes. This ensures the entire command is treated as a single string argument for the remote shell.

bash
# Failed attempt (simplified)
ssh user@host <<EOF
  docker exec -i my-db-container psql -U user -d db <<SQL
    INSERT INTO crawl_jobs ...;
    DELETE FROM tainted_docs ...;
SQL
EOF

# Workaround: Single-line -c flag with escaped quotes
ssh user@host "docker exec my-db-container psql -U user -d db -c 'INSERT INTO crawl_jobs ...; DELETE FROM tainted_docs ...;'"

Always test your production-bound shell scripts in a staging environment first!

3. The Perils of Duplication

A Minor Note, but Critical for Tech Debt: We noticed globals.css has duplicate scrollbar utilities: .no-scrollbar (line 108) and .scrollbar-none (line 346). While not a blocking issue, it's a clear indicator of tech debt that needs to be consolidated for maintainability and bundle size. It's a reminder that even small inconsistencies can accumulate.

What's Next?

With the deployment healthy and the immediate goals achieved, our next steps are clear:

  1. Re-crawl BetrVG: Verify clean text extraction for the documents that were previously tainted.
  2. Real-Device Mobile Testing: This is paramount. No emulator can fully replicate the nuances of a physical device, across various screen sizes and operating systems.
  3. RLS Policy for crawl_jobs: Implement Row-Level Security for our new crawl_jobs table to ensure data integrity and access control.
  4. Consolidate Scrollbar Utilities: Tackle that minor tech debt in globals.css.
  5. Touch Target Polish Pass: A follow-up to ensure all interactive elements meet the recommended 44px minimum touch target size for optimal mobile usability.

This session was a fantastic example of the multi-faceted nature of modern web development. From backend data integrity to pixel-perfect mobile UIs, every piece plays a role in delivering a robust and user-friendly experience. We're excited to see the impact of these changes in the wild!

json
{"thingsDone":[
  "Fixed HTML stripping bug in document-processor.ts (prefers local file)",
  "Fixed crawler mimeType text/html -> text/plain bug",
  "Deleted 153 tainted BetrVG docs and created crawl_jobs table on production",
  "Implemented mobile-first CSS overhaul (viewport, sidebar, dialogs, tables, layouts)",
  "Refined mobile navigation ('Style' -> 'Memory' + icon)",
  "Fixed sidebar component reusability issue within Sheet on mobile",
  "Polished mobile sidebar (hide collapse/heartbeat, logo always visible)"
],"pains":[
  "Sidebar component hidden inside Sheet due to conflicting responsive utility classes",
  "SSH heredoc failing to pass through docker exec for multi-line SQL commands",
  "Duplicate scrollbar utilities in globals.css (tech debt)"
],"successes":[
  "All planned features and fixes deployed to production",
  "Significant improvement in mobile user experience",
  "Critical crawler bugs resolved, enabling clean data ingestion",
  "Learned practical lessons on component reusability and shell scripting"
],"techStack":[
  "TypeScript",
  "Node.js",
  "PostgreSQL",
  "Docker",
  "Tailwind CSS",
  "React",
  "Next.js"
]}