← nyxcore-systems
5 min read

Unlocking Cross-Repo Clarity: Our 10-Step AI-Powered Integration Analysis Workflow Goes Live

Dive into the journey of designing, implementing, and deploying a robust 10-step AI-powered workflow for cross-repository integration analysis, navigating critical design pivots and battling LLM token limits along the way.

LLMWorkflowAutomationSoftwareArchitectureDevOpsAIGeminiCodeReviewIntegrationSystemDesign

The modern software landscape is a sprawling web of interconnected services, often spanning multiple repositories. Understanding these intricate cross-repo integrations manually can be a monumental task, prone to errors and overlooked dependencies. That's why we embarked on a mission to automate this discovery process, culminating in the successful deployment of our new Integration Analysis workflow template.

This isn't just any workflow; it's a sophisticated, 10-step pipeline designed to unearth deep integration insights using a combination of advanced AI modules, including our internal "Ipcha Mistabra" for deep contextual analysis and "Cael hardening" for robust security scrutiny.

From Concept to Production: The Journey of a 10-Step Pipeline

Our goal was ambitious: design, implement, and deploy a comprehensive workflow that could autonomously map, analyze, and even secure cross-repository integrations. The journey began with intense brainstorming sessions, where we meticulously defined each of the ten critical steps. These steps, now enshrined in our src/lib/constants.ts (a file that grew by a thousand lines!), cover everything from initial surface discovery to in-depth security and ethical analysis.

A key part of our process involved documenting the entire design in docs/plans/2026-03-08-integration-analysis-design.md, ensuring a clear blueprint for implementation.

The Critical Code Review Pivot

During our code review, a critical flaw emerged: our initial design for intIpchaChallenge meant that review steps couldn't effectively leverage compareProviders. This was a significant blocker, as comparing outputs from multiple LLMs is crucial for robust analysis and selecting the best alternative.

The fix involved a crucial architectural split:

  • We separated intIpchaChallenge into intIpchaAnalysis (the LLM-driven deep dive) and intIpchaReview (the human oversight step). This allowed us to apply compareProviders where it truly mattered: in the automated analysis phase, presenting human reviewers with refined, multi-perspective insights.
  • We also replaced the impossible providerFanOutConfig (which our engine only supports for llm steps) with compareProviders throughout the pipeline, allowing users to pick the best alternative from several LLM suggestions. This ensures flexibility and quality control.
  • For steps requiring multiple distinct outputs, like our fan-out analysis on integration categories, we implemented a clever solution: explicit ### N. output formatting within the LLM prompt. This structured output ensures that subsequent steps can correctly parse and process each sub-output individually.

With these changes, committed as 9e36dd2, we pushed our initial implementation to production. The moment of truth arrived with our first real workflow run, analyzing the integration between CodeMCP and nyxcore-systems. Workflow b6947b7a completed successfully – a huge milestone!

Navigating the AI Frontier: Lessons Learned from the "Pain Log"

Building with LLMs is exhilarating, but not without its unique challenges. Our journey provided some valuable lessons:

1. When the Engine Says No: providerFanOutConfig vs. compareProviders

  • The Idea: Initially, we wanted to use providerFanOutConfig on certain StepTemplate instances to automatically generate multiple parallel outputs from different LLMs.
  • The Reality: We quickly discovered that our workflow engine only supports providerFanOutConfig on explicit llm steps. It wasn't designed for a general StepTemplate interface.
  • The Pivot: Instead, we leveraged compareProviders. This allows us to run multiple LLMs concurrently and then present their diverse outputs as alternatives, enabling a user or a subsequent automated step to select the best one. It’s a slightly different pattern but achieves the goal of multi-perspective analysis.

2. Battling the Beast: LLM Token Truncation

  • The Problem: During our first production run, we observed a critical issue: Google Gemini, with maxTokens set to 8192, truncated its output to a mere 328 completion tokens on steps with large input contexts (specifically intSecurityAnalysis and intIpchaAnalysis). This meant incomplete or missing analysis – unacceptable for a critical workflow.
  • The Fix: We immediately diagnosed this as an LLM token limit issue. The solution was straightforward: we bumped the maxTokens from 8192 to 16384 for intRecon, intSecurityAnalysis, and intIpchaAnalysis. Step 1 (Surface Discovery) was fine at 8K, suggesting the context size varies significantly across steps.
  • Ongoing Monitoring: This highlights a crucial point for anyone building with LLMs: maxTokens limits are real, and they can silently cripple your analysis. We're now monitoring our Gemini Flash usage closely, as even higher limits might be needed for extremely large repositories.

3. Semantic Tagging: An Area for Future Enhancement

  • We noted that insightScope: "ethic" isn't automatically tagged via the template path. Our insight-persistence.ts currently looks for workflow names containing "Ipcha Mistabra" or having providerFanOutConfig. Integration Analysis workflows, as currently configured, won't trigger this specific ethical tagging. This is a clear opportunity to enhance our StepTemplate with explicit insightScope support for richer, more accurate metadata.

The Current State: Deployed, Verified, and Ready

As of now, our Integration Analysis workflow is fully deployed to production with the token limit fix (commit 34d6b8c). The first successful run (workflow b6947b7a-7b36-4653-947d-e8b2f18bf6b9) proved its capability, analyzing the CodeMCP ↔ nyxcore-systems integration end-to-end. All 10 steps completed, with alternatives correctly generated on Steps 1, 2, 6, 7, and 9. Critically, the fan-out on Step 4 successfully produced 6 distinct sub-outputs, one for each integration category.

We're leveraging a diverse set of models, including claude-sonnet-4-20250514, gemini-2.5-flash, and gpt-4o-mini, to ensure comprehensive and varied insights.

What's Next? Continuous Improvement

Our work isn't done. We're already looking at the immediate next steps:

  1. Verification: Re-running the workflow with the raised token limits to fully verify that Gemini now produces complete outputs.
  2. Model Exploration: Considering gemini-2.5-pro for steps requiring even deeper and more nuanced analysis, potentially replacing flash in critical areas.
  3. Platform Enhancement: Adding explicit insightScope support directly to StepTemplate for more granular and automated ethical tagging.
  4. Feature Expansion: Exploring the extension of StepTemplate to natively support providerFanOutConfig and dualProviderAutoSelect for even more powerful and flexible multi-LLM orchestration.
  5. Quality Assurance: A thorough review of the quality of fan-out outputs to ensure the ### N. splitting mechanism is consistently delivering high-quality, actionable insights.

This journey has been a testament to the power of automated workflows and the adaptability required when building with cutting-edge AI. We're excited about the clarity and efficiency this new workflow brings to understanding our complex system integrations.