From Crawler Quirks to Mobile Polish: Shipping a Smoother User Experience

Just wrapped up a significant development sprint, and what a journey it's been! This past week was a whirlwind of debugging, designing, and deploying, culminating in a production release that not only squashes some critical data integrity issues but also delivers a much-anticipated mobile-first experience.

Let's pull back the curtain on the challenges we faced and the solutions we implemented.

Mission Brief: The Crawler Cleanup

Our application relies heavily on accurate data ingestion, and lately, we'd been seeing some peculiar behavior. Documents weren't always rendering as expected, and a deeper investigation revealed two core issues plaguing our crawler.

HTML Stripping & MimeType Mischief

The first culprit was a subtle but critical bug in our document-processor.ts. It turned out that our processor was sometimes re-fetching the source URL even when a local file was available, leading to inconsistent HTML stripping. The fix (0957b1b) was to ensure we prioritize the local file, preserving the original content integrity.

Simultaneously, we discovered our site-crawler-service.ts was misclassifying text/html content as text/plain. This meant our beautifully structured web pages were being treated as raw text, leading to a mangled mess in our database. Correcting the mimeType handling at line 333 was crucial for proper content interpretation.

Data Hygiene: A Necessary Evil

With the crawler bugs identified and fixed, we had a backlog of "tainted" documents – specifically, 153 BetrVG (German Works Constitution Act) documents that had been incorrectly processed. There was no way around it: these needed to be deleted and re-crawled. This required a manual SQL intervention to create a crawl_jobs table on production, preparing the ground for a fresh, clean ingestion. It's a stark reminder that sometimes, the most robust fixes involve a bit of manual data surgery.

Embracing Mobile-First: A UI Transformation

While the crawler fixes were critical under-the-hood work, the most visible changes landed on the user interface, specifically for mobile users. Our goal was not just to make the site "work" on mobile, but to make it shine. This meant a full mobile-first CSS overhaul, driven by a series of focused tasks.

Crafting a Responsive Viewport

The first step was to refine the viewport meta tag (4d43396). We removed maximumScale:1 to allow users full control over zooming, and importantly, added viewportFit:cover along with safe-area padding. This ensures our UI gracefully handles modern phone notches and rounded corners, providing a truly edge-to-edge experience without content being cut off.

html

<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">

Sidebar Reimagined: From Vertical to Horizontal

One of the most significant changes was transforming our traditional desktop sidebar into a mobile-friendly horizontal scrollable tab bar (7aee355). Using Tailwind CSS's responsive utilities (md:hidden / hidden md:block), we created a seamless transition, giving mobile users easy access to navigation items without consuming precious vertical screen space.

Dialogs That Fit: Responsive Modals

Our application uses various dialogs, and on smaller screens, they often felt cramped or forced. We addressed this by implementing mobile-specific margins (w-[calc(100%-2rem)]), padding (p-4 md:p-6), and ensuring they were vertically scrollable (max-h-[85vh] overflow-y-auto). This makes interacting with modals on a phone a much more pleasant experience (1fc31a3).

Smarter Tables: Hiding Complexity

Complex data tables are notoriously difficult on mobile. For our model usage table, we strategically hid less critical columns on smaller screens (hidden md:table-cell on 4 columns) and introduced a "tap-to-expand" pattern for rows. This allows users to focus on key information initially, with the option to reveal more details as needed (97861ee).

Flexible Layouts: Adapting Content Blocks

Across several detail pages (Ipcha, Projects), we refactored layout elements to be more flexible. Instead of rigid horizontal arrangements, elements now stack vertically on small screens and transition to horizontal on larger ones (flex-col gap-2 sm:flex-row), ensuring content is always readable and well-organized (07817b5).

Polishing Mobile Navigation

Beyond the layout, we also refined the mobile navigation experience. The "Style" option was rebranded to "Memory" and given a distinct Brain icon (19ea037), making its purpose clearer and more intuitive. Small touches like these significantly improve usability.

Finally, a persistent bug where our sidebar sheet wasn't displaying correctly on mobile was traced back to a CSS specificity issue. By adding a className prop to our Sidebar component and having the Sheet component pass className="flex w-full" to override the hidden utility, we ensured it always renders as intended (19ea037). Further polish included hiding the collapse toggle and heartbeat indicator on mobile, keeping the logo always visible (bcdb179), creating a clean, focused mobile sidebar experience.

Lessons from the Trenches: Challenges & Solutions

No significant development effort is without its snags. Here's a look at some of the "pain points" and how we navigated them:

The Elusive Sidebar in a Sheet

Challenge: We wanted to reuse our <aside> sidebar component inside a mobile Sheet component. However, the hidden md:flex utility on the <aside> meant it was invisible even within the Sheet below md breakpoints. The hidden was overriding the Sheet's display properties.

Solution: The workaround involved adding a className prop to the Sidebar component. The Sheet now explicitly passes className="flex w-full" to the Sidebar, which overrides the hidden utility and ensures it's always visible when the sheet is open. This highlights the importance of understanding CSS specificity and providing escape hatches for component overrides.

SSH Heredoc vs. Docker Exec

Challenge: When performing the manual SQL operations for the crawl_jobs table, I initially tried to use an SSH heredoc to execute a multi-line SQL script via docker exec psql. This failed because heredocs don't pass through SSH and docker exec as expected, leading to syntax errors or incomplete commands.

Solution: The workaround was to condense the SQL command into a single line and use the -c flag for psql, carefully escaping quotes. This is a common pattern for executing commands remotely or within containers and a good reminder to keep commands concise for such scenarios.

bash

ssh user@server "docker exec -i my-container psql -U user -d db_name -c 'CREATE TABLE crawl_jobs (id SERIAL PRIMARY KEY, url TEXT NOT NULL, status TEXT DEFAULT \"pending\");'"

Note: On a minor housekeeping front, our globals.css currently has duplicate scrollbar utilities (.no-scrollbar and .scrollbar-none). This will be consolidated in a future cleanup pass.

The Road Ahead: Immediate Next Steps

With these changes live, the system is deployed and healthy! But the work doesn't stop here. Our immediate next steps include:

Re-crawl BetrVG documents: Verify clean text extraction now that the crawler bugs are fixed.
Real-device mobile testing: Thoroughly test the new mobile UI across various devices and screen sizes.
Add RLS policy for crawl_jobs table: Secure the new table with appropriate Row-Level Security.
Consolidate scrollbar utilities: Clean up globals.css by merging .no-scrollbar and .scrollbar-none.
Touch target polish pass: Ensure all interactive elements meet the recommended 44px minimum touch target size for optimal mobile usability.

It's incredibly satisfying to see these significant improvements go live. This release not only makes our data more reliable but also dramatically enhances the user experience for everyone on mobile. Onward to the next challenge!