← nyxcore-systems
5 min read

Taming the Prompt Beast: How We Slashed AI Context Bloat with Step Digests and Project Wisdom

We tackled a whopping 374KB prompt cascade in our Deep Build Pipeline by introducing intelligent step digests and a project-centric context system, making our AI workflows leaner and smarter.

LLMPrompt EngineeringContext CompressionWorkflow EnginePrismaTypeScriptAIDeveloper ToolsClaude Haiku

In the world of AI-driven development, context is king – but too much context can bring even the mightiest LLMs to their knees. We recently faced this exact challenge head-on with our Deep Build Pipeline, where a "prompt cascade" had swelled to an unwieldy 374KB. Imagine feeding an AI model a novel's worth of text for every single step! It's slow, expensive, and often dilutes the very focus we're trying to achieve.

Our mission: compress the noise, elevate the signal. We set out to achieve this through a two-pronged approach: intelligent step digests and a project-centric context system. The goal was to provide our workflow engine with just the right amount of information, precisely when it needed it.

The Quest for Leaner Prompts: Introducing Step Digests

The root of our prompt bloat was simple: our workflow engine was passing the entire output of previous steps into subsequent prompts. While comprehensive, this quickly became redundant, especially for complex, multi-stage pipelines.

Our solution? Step Digests. Instead of full outputs, we now generate concise summaries for completed workflow steps, using an LLM to distill the essence.

How We Implemented It:

  1. Schema Evolution: We added a new digest field to our WorkflowStep model in prisma/schema.prisma. This field is designed to hold the summarized text.

    prisma
    model WorkflowStep {
  // ... existing fields ...
  digest String? @db.Text // New field for context compression
}

  2. The Digest Service: We built src/server/services/step-digest.ts, housing a generateStepDigest() function. This function uses claude-haiku-4-5-20251001 (our trusty Haiku model) to create a summary. We configured it with a 1024 max token limit and a smart skip logic: if an output is already less than 2000 characters, we don't bother summarizing it. On failure, we gracefully fall back to truncation to ensure some context is always available.

  3. Engine Integration: The core intelligence resides in src/server/services/workflow-engine.ts:

    • We enhanced our ChainContext interface to include stepDigests: Map<string, string> and projectWisdom: string.
    • When a workflow resumes, buildChainContext() now populates stepDigests from the digest fields of previously completed steps.
    • Crucially, after a step successfully completes, we call generateStepDigest() and persist the result both in the database and the active chain context.
    • We introduced a new template variable: {{steps.Label.digest}}. This allows prompt engineers to explicitly request the summarized version of a step's output, with a built-in truncated-content fallback.

  4. Targeted Application: We updated src/lib/constants.ts to strategically leverage .digest in our most context-hungry deep pipeline steps:

    • deepFeatures (Add Features): Now uses Idea + Research digests.
    • deepReview1 (Review): Uses Research digests.
    • deepExtend (Extend & Improve): Leverages Add Features + Research + Idea digests.
    • deepWisdom (Project Wisdom): Uses Idea + Research digests.
    • deepImprove (Improve): Benefits from Extend & Improve digests.

This targeted approach ensures that only the most relevant, distilled information is passed to the LLM, drastically reducing prompt size without losing critical context.

Elevating Context: The Power of Project Wisdom

While step digests tackle the micro-level context bloat, we also identified a need for macro-level, overarching context – Project Wisdom. Workflows often operate in isolation, yet they're almost always part of a larger project. Imagine an AI agent trying to generate code for a new feature without understanding the project's existing consolidation patterns, coding standards, or overall goals.

Our solution was to introduce a first-class Project entity and link our workflows and repositories to it.

How We Implemented It:

  1. Project-Centric Schema: We updated prisma/schema.prisma to link Workflow and Repository models to a Project via a projectId field. This establishes clear, relational ties.

    prisma
    model Workflow {
  // ... existing fields ...
  projectId String? @db.Uuid
  project   Project? @relation(fields: [projectId], references: [id])
}

model Repository {
  // ... existing fields ...
  projectId String? @db.Uuid
  project   Project? @relation(fields: [projectId], references: [id])
}

model Project {
  id          String       @id @default(uuid()) @db.Uuid
  name        String
  workflows   Workflow[]
  repositories Repository[]
  // ... other project fields ...
}

  2. TRPC Router Updates: Our src/server/trpc/routers/workflows.ts was updated to handle project relationships seamlessly:

    • create and update mutations now accept projectId to connect/disconnect workflows.
    • The get query includes project: { select: { id, name } } to fetch linked project details.
    • duplicate mutation now preserves the projectId.
    • We added a new byProject query, allowing us to filter workflows based on their associated project.

  3. Loading Project Wisdom: A new loadProjectWisdom() function in workflow-engine.ts is responsible for aggregating project-level context. This currently loads consolidation patterns (e.g., how to merge code) and code analysis patterns from all repositories linked to the project. This "wisdom" is then made available via the {{project.wisdom}} template variable.

  4. UI Integration: To make this tangible for users, we updated src/app/(dashboard)/dashboard/workflows/[id]/page.tsx. Now, within the workflow settings panel, there's a <select> dropdown to link a workflow to an existing project. A FolderOpen icon clearly indicates a linked project, and hint text educates users about the magic of {{project.wisdom}} auto-injection.

Under the Hood: Infrastructure & Smooth Sailing

As part of these changes, we ran npm run db:push && npm run db:generate to sync our Prisma schema and regenerate the client. This ensured our application code was perfectly aligned with the new database structure, which now includes workflow_steps.digest, workflows.project_id, and repositories.project_id.

Remarkably, this development session was incredibly smooth. We encountered no major issues related to the new features. The only "pain" logged was a pre-existing type error in src/app/(dashboard)/dashboard/discussions/[id]/page.tsx:139 (a Badge variant issue), which was unrelated to this work and is on our backlog.

What's Next: Verifying Our Gains

With the code committed and type-checked, our immediate next steps are focused on verification:

  1. Push these changes to origin.
  2. Run a full workflow end-to-end and use Prisma Studio to confirm that digest fields are correctly generated and stored.
  3. Crucially, compare the resolved prompt size for a deep pipeline step (e.g., step 9) before and after these changes. Our target is a >60% reduction!
  4. Test the {{project.wisdom}} variable by linking a workflow to a project that has active consolidation data.
  5. Consider adding RLS (Row Level Security) policies for the new projectId columns if cross-tenant access becomes a concern.
  6. Finally, we'll tackle that pesky pre-existing type error in the discussions page.

By intelligently compressing context and providing a higher-level project understanding, we're making our AI workflows more efficient, more focused, and ultimately, more powerful. This is a significant step towards building truly scalable and intelligent automated systems.