From Protocol to Pixels: Our Latest Sprint Delivers IPCHA & Vision for Visual Notes

What a sprint it's been! Our recent development session was a whirlwind of activity, culminating in the deployment of a critical new protocol and setting the stage for a game-changing feature that will bring visual intelligence to our project notes. Let's unpack the highlights.

The IPCHA Protocol: A New Foundation for Trust

At the heart of this sprint was the full implementation and deployment of the IPCHA protocol. This isn't just another module; it's a foundational layer designed to enhance fairness and robustness within our system's arbitration and claims mechanisms.

We poured significant effort into crafting this core component:

• Comprehensive Implementation: We developed 15 distinct modules, ensuring every facet of the protocol was meticulously coded.
• Rigorous Testing: With 78 dedicated Python tests spanning ipcha/, src/arbitration/, benchmarks/, and sdrl_claims/, we've built a high degree of confidence in its stability and correctness.
• Production Ready: The entire protocol is now committed on main (specifically b34b44f) and fully deployed across our production environment. This marks a significant milestone in our system's evolution.

Bringing IPCHA to Life: API & Dashboard

A powerful protocol is only as useful as its accessibility. To make IPCHA consumable and manageable, we built out a comprehensive API service and an interactive dashboard. This involved a full-stack effort:

• FastAPI Sidecar: A dedicated FastAPI service (ipcha/api.py) now runs as a sidecar, exposing 10 robust endpoints on port 8100. This provides a direct, high-performance interface to IPCHA's capabilities.
• Secure REST Proxy: For seamless integration with our existing application, we implemented a REST proxy (src/app/api/v1/ipcha/) with 11 endpoints, secured by nyx_ip_ token authentication. This ensures that only authorized services can interact with the protocol.
• Data Persistence & Security: We leveraged Prisma to define new models—IpchaApiToken, IpchaUsageLog, and IpchaJob—complete with Row Level Security (RLS) to safeguard sensitive data and usage patterns.
• Full-Stack Integration: Our frontend now features a dedicated token service, a sidecar client, and a tRPC router, all feeding into an intuitive 8-tab dashboard. This dashboard provides real-time insights and control over IPCHA operations.
• BYOK (Bring Your Own Key) Integration: A key feature for flexibility, we've enabled BYOK integration for LLM providers via the X-LLM-Api-Key header and a ProviderModelPicker. This empowers users to leverage their preferred LLM services directly.

The success of this integration was thoroughly validated: PR #133 is merged, deployed on main, and all 14 integration tests passed with flying colors. You can find the detailed report in docs/reports/2026-03-15-ipcha-api-integration-test.md.

Vision for the Future: Image Uploads in Project Notes

While the IPCHA deployment was a major win, we also spent valuable time brainstorming and prototyping an exciting new feature: image uploads for Project Notes with LLM-powered descriptions.

Current project notes are text-heavy, but often, a screenshot, a diagram, or a whiteboard sketch can convey information far more effectively. The challenge is making these images searchable and actionable. Our solution?

• The "Describe-on-Upload" Strategy: We've decided on Option A: When an image is uploaded, an LLM will immediately analyze and describe its content. This rich text description will then be stored in the MemoryEntry, making visual information fully searchable and understandable by our AI systems.

• Proof of Concept Success: The early results are incredibly promising! We tested with three annotated screenshots, and the LLM successfully identified:

  • Specific UI elements (pages, fields, values, statistics).
  • Handwritten annotations (colored circles, arrows, text in various colors).
  • Cross-references between different images (e.g., "feature checklist" linking to a "settings page" and "data sources").
  • Actionable items derived directly from the annotations.

• Building on Existing Infrastructure: We're not starting from scratch. Our src/server/services/storage.ts adapter already handles local and S3 storage for JPEG, PNG, and WebP files up to 50MB. The immediate next steps involve adding vision support to our existing LLM adapters (Anthropic or OpenAI) and extending our MemoryEntry model to include an imageKey field.

This feature is a major leap towards creating more comprehensive, intelligent, and visually rich project documentation.

Lessons Learned & Minor Challenges

No development sprint is without its bumps in the road. While the IPCHA deployment went smoothly, we acknowledge that the initial pain points encountered during its development are thoroughly documented in letter_20260315_0001.md.

One active item from this session is a user-reported bug: the "Analysis feature on projects 'hängt manchmal'" (sometimes hangs). This is on our radar and will be investigated promptly to ensure a smooth user experience. These minor issues are critical for continuous improvement and maintaining the quality of our product.

What's Next? Paving the Road Ahead

With IPCHA deployed and the image upload feature designed, our immediate focus shifts to bringing that visual intelligence to life:

1. Build Image Upload for Notes: This is our top priority. We'll finalize the design spec, then implement the imageKey field in MemoryEntry, add vision support to at least one LLM adapter, and create the drag-and-drop/file picker UI on the Notes page. The goal is to store the image, send it to the vision LLM, and save the description as the note's content, with a thumbnail shown in the note card.
2. Enhance Project Onboarding: We'll immediately leverage the new image upload feature to create a rich Project Onboarding note, using those three annotated screenshots from our PoC.
3. Actionable Onboarding: From this visual onboarding note, we'll convert key insights into concrete action points:
   • Refactor sources (e.g., "Letters menüpunkt," Git filesystem view).
   • Implement auto-run analysis on onboarding and debug the hanging issue.
   • Auto-fill settings from README files and integrate a branch selector.
   • Refine the onboarding flow for existing vs. new repositories.
   • Create demo projects for Claude Code one-shot execution.
4. Generate Implementation Prompts: We'll then use these action points to generate detailed implementation prompts for our development workflow.
5. Minor IPCHA Cleanup: A few final touches include adding an env_file for Docker Compose and ensuring idempotent RLS SQL for robustness.

This session was a powerful blend of solidifying core infrastructure and innovating for future user experience. We're excited about the capabilities IPCHA brings and even more thrilled about the potential of visual AI in transforming how we capture and utilize project insights. Stay tuned for more updates!