Unlocking Vision: Building LLM-Powered Image Uploads for Project Notes (and a Production Scare!)

Just wrapped up a sprint that felt like a mini-marathon, and the reward is sweet: image uploads for Project Notes, supercharged with LLM vision for automatic descriptions. This wasn't just about dropping a file; it was about integrating visual context directly into our LLM-powered workflows, making our system smarter and our users' lives easier.

But as with any ambitious feature, the path was paved with both elegant solutions and a few "lessons learned the hard way." Let's dive into what we built and what we stumbled over.

The Vision: More Than Just Pixels

Our goal was clear: allow users to upload images to their project notes, have an LLM automatically describe those images, and then make those descriptions available within our existing workflow engine via a {{notes}} template variable. On top of that, we wanted users to pick which LLM provider and model described their images.

This wasn't just a UI feature; it was about enriching the data our AI agents interact with. Imagine uploading a screenshot of an error, and the system automatically generates a detailed description, which can then be fed into a "Debug Issue" workflow. Powerful stuff.

Building Blocks for Visual Intelligence

Here's how we pieced it together:

Data Persistence with Prisma: First, we needed to store references to the images and their descriptions. This meant extending our MemoryEntry and ProjectNote models in Prisma:
typescript
```
model ProjectNote {
  id             String    @id @default(uuid())
  // ... other fields
  imageKey       String?   // S3/local storage key
  imageDescription String? // LLM-generated description
}
```
This simple addition laid the groundwork for attaching visual context.
LLM Vision: Speaking to the Machines: The core challenge here was integrating with multiple LLM providers (OpenAI, Anthropic) for their vision capabilities, each with its own API quirks.

We extended our LLMMessage.content type to a MessageContent union, allowing for both plain text and structured blocks:
typescript
```
type TextBlock = { type: 'text'; text: string; };
type ImageBlock = { type: 'image_url'; image_url: { url: string; detail?: 'auto' | 'low' | 'high'; }; };
type MessageContent = string | Array<TextBlock | ImageBlock>;
```
Our LLMService adapters then handle the translation:
- OpenAI adapter: Maps ImageBlock to their image_url format, often using base64 data URIs for direct uploads.
- Anthropic adapter: Maps ImageBlock to its native format, which can handle direct URLs or base64. We also added getTextContent() helpers for extracting just the text from complex messages, useful for non-vision-enabled LLMs or specific contexts.
Flexible Storage: To handle image files, we extended our StorageAdapter interface with getFileBuffer(key). This keeps our storage layer agnostic, whether we're using local disk for development or S3 in production. Both implementations were updated to support retrieving file buffers for LLM processing.
The image-describe Service: This is the brains of the operation. It orchestrates the entire process:
- Takes an imageKey, tenantId, userId, and the user-selected provider and model.
- Retrieves the image buffer using the StorageAdapter.
- Constructs the appropriate ImageBlock for the chosen LLM.
- Calls the LLM adapter to get a description.
- Returns the descriptive text.
This service is a clean abstraction, making it easy to swap LLM providers or add new image processing steps in the future.
API & UI: Bringing it to Life: We added new tRPC mutations (getImageUploadUrl, describeImage) to both our memory and projects.notes routers. This allows the frontend to request a pre-signed URL for direct image uploads (for efficiency) and then trigger the description process.

On the UI side, the NotesTab in the project dashboard now sports a sleek drag-and-drop upload area. Crucially, it includes a ProviderModelPicker, giving users control over which LLM describes their images, with openai/gpt-4o-mini as a sensible default. Once uploaded, a thumbnail appears, and the auto-generated description populates, ready for editing.
Workflow Integration with {{notes}}: This is where the magic truly happens. We implemented loadNotesContent() to aggregate all notes (including their image descriptions) for a given project. Then, we registered {{notes}} as a template variable in our workflow-engine.ts, allowing users to inject the full context of their project notes, visual and textual, directly into any LLM prompt.

This means a workflow can now "see" the images associated with a project, understand their content, and act upon them.

Lessons from the Trenches: The `pgvector` Scare

No significant feature ships without a few battle scars. This sprint gifted us a particularly memorable one.

The `db push` Debacle

In a moment of hurried deployment, I made a critical error: I ran npx prisma db push --accept-data-loss directly on our production database.

The result? Prisma, seeing our embedding vector(1536) column (used by pgvector for vector embeddings) as an "unsupported" type, decided it was an "extra" column and promptly dropped it.

Panic. Immediate, cold-sweat panic. Our workflow_insights table, crucial for semantic search and RAG, was suddenly missing its core.

The Fix (and the Lesson): Thankfully, the data itself wasn't lost, just the column definition. I was able to restore it with raw SQL:

sql

ALTER TABLE workflow_insights ADD COLUMN IF NOT EXISTS embedding vector(1536);
CREATE INDEX IF NOT EXISTS workflow_insights_embedding_idx ON workflow_insights USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);

This was a stark reminder of a lesson already documented but violated in the heat of the moment: NEVER use db push on production. Always, always, always use proper migration scripts (like our ./scripts/db-migrate-safe.sh) that generate explicit ALTER TABLE statements. Prisma's db push is a development tool, not a production deployment mechanism for databases with "unsupported" column types.

User-Centric Development: The Right Place for the Feature

Another minor, but common, "pain" point was initially building the image upload feature on the general Memory Hub page. It seemed logical from a "memory" perspective. However, user feedback quickly clarified: the most valuable context for images was within specific Project Notes.

This led to adding the imageKey and imageDescription to the ProjectNote model (in addition to MemoryEntry) and adapting the tRPC mutations to the projects.notes sub-router. It's a classic example of how even well-intentioned architectural decisions can miss the mark if not directly aligned with user workflow. Always build where the user needs it most.

What's Next?

With image uploads live and enriching our data, the immediate next steps involve dogfooding:

Uploading annotated screenshots of our current features.
Using our own "Enrich" workflows to convert those descriptions into actionable points.
Generating implementation prompts for upcoming features like Project Onboarding.

This image upload capability is a significant step towards a truly intelligent assistant, making our project notes richer and our LLM workflows more powerful. It was a challenging but rewarding sprint, and we learned a few critical lessons along the way. Now, onto the next adventure!

json

{
  "thingsDone": [
    "Image Upload for Project Notes (full implementation)",
    "Prisma schema updated with imageKey and imageDescription",
    "LLM Vision types extended for multi-modal content",
    "OpenAI and Anthropic LLM adapters updated for vision",
    "Storage extension with getFileBuffer(key)",
    "Image describe service created",
    "Notes content loader for {{notes}} template variable",
    "Upload API routes for notes images",
    "tRPC mutations for image upload and description",
    "Workflow engine registered {{notes}} variable",
    "UI for drag-and-drop upload, ProviderModelPicker, auto-describe, thumbnails",
    "Critical pgvector column restored after accidental drop",
    "Design spec and implementation plan documented"
  ],
  "pains": [
    "Accidental dropping of pgvector column on production using `db push --accept-data-loss`",
    "Initial misplacement of image upload feature (Memory Hub vs. Project Notes)"
  ],
  "successes": [
    "Seamless LLM vision integration across multiple providers",
    "Robust image storage and retrieval system",
    "Intuitive drag-and-drop UI for image uploads",
    "Powerful workflow integration with {{notes}} template variable",
    "Quick recovery from production database incident",
    "User-driven feature placement correction"
  ],
  "techStack": [
    "Next.js",
    "tRPC",
    "Prisma",
    "PostgreSQL",
    "pgvector",
    "OpenAI API",
    "Anthropic API",
    "S3 (or Local Storage)",
    "TypeScript",
    "Docker"
  ]
}