← nyxcore-systems
5 min read

Beyond Unit Tests: Unlocking Deeper Code Quality with Ipcha's Repo-Level Audits

Discover how we built Ipcha's new repository-level auditing and stress-testing system, introducing intelligent file prioritization and a tiered workflow to ensure robust code quality at scale.

code-qualityauditingsoftware-developmentdevopsAIworkflow-automationtypescriptprisma

Maintaining a robust, high-quality codebase is a perpetual challenge. As projects grow, the sheer volume of code can make it difficult to catch regressions, ensure consistency, and proactively identify areas prone to issues. At Ipcha, our mission is to empower developers with intelligent self-testing systems, and today, we're thrilled to pull back the curtain on a significant leap forward: repo-level auditing and stress testing.

This past week, our team embarked on an ambitious journey to equip Ipcha with the capability to not just test individual components, but to intelligently scan, prioritize, and deep-dive into entire repositories. The goal was clear: provide a comprehensive, automated safety net that goes beyond traditional unit tests, ensuring our codebases remain resilient and reliable.

The Challenge: Auditing a Universe of Code

Imagine a vast galaxy of code files. How do you decide which stars to examine closely? Auditing an entire repository isn't just about running tests on every file; it's about smart resource allocation. We needed a system that could:

  1. Discover: Identify all relevant files within a given repository.
  2. Prioritize: Focus testing efforts on files most likely to introduce regressions or harbor existing issues.
  3. Scale: Handle large codebases efficiently, breaking down complex audits into manageable, actionable units.
  4. Report: Provide clear, actionable insights back to developers.

Ipcha's Solution: Intelligent Prioritization and Tiered Auditing

Our answer came in a multi-pronged approach, anchored by a new file prioritization system and a tiered auditing workflow.

The Intelligent File Prioritizer

To avoid the "boil the ocean" problem, we developed a sophisticated file-prioritizer.ts service. This service assigns a weighted score to each file, helping Ipcha decide where to direct its most intensive auditing efforts. We considered several critical factors:

  • Churn (0.35 weight): Files that change frequently are often hotspots for new bugs.
  • Size (0.25 weight): Larger files tend to be more complex and harder to reason about, increasing their risk profile.
  • Imports (0.20 weight): Files with many dependencies or that are heavily depended upon (imports) indicate high coupling and potential for cascading failures.
  • Staleness (0.20 weight): Files that haven't been touched in a while might contain outdated logic or hidden vulnerabilities that recent changes haven't exposed.

By combining these factors, Ipcha can intelligently focus on the riskiest parts of your codebase first, maximizing the impact of each audit cycle.

The Tiered Auditing Workflow

Auditing isn't a one-size-fits-all process. We designed a two-tier system to provide both broad coverage and deep investigation:

  • Tier 1: The Broad Scan (Discovery & Grouping)

    • Ipcha's repo-audit-service.ts takes your repository configuration (project picker, glob patterns for inclusion/exclusion) and performs an initial discovery phase.
    • It uses the file prioritizer to score files and groups them, identifying potential areas of concern. This tier acts as a high-level overview, quickly flagging files that warrant closer inspection.
    • The results of this tier feed into the next.

  • Tier 2: The Deep Dive (Per-File Workflows)

    • The workflow-engine.ts then takes the flagged files from Tier 1.
    • For each file that crosses a defined risk threshold, it dynamically spawns individual, per-file auditing workflows. These workflows can involve more intensive testing, static analysis, or even AI-driven code reviews tailored to the specific file's context.
    • This ensures that critical files receive the in-depth scrutiny they deserve without overwhelming the system with unnecessary checks on low-risk files.

Under the Hood: Key Implementation Highlights

Bringing this vision to life involved touching several core components of the Ipcha system. Here's a glimpse at the key architectural changes:

  • Schema Evolution: We introduced three crucial columns to our audit_runs table: tier (to distinguish between T1 and T2 runs), filePath (for T2 runs targeting specific files), and parentRunId (to link deep-dive T2 runs back to their originating T1 audit).
    sql
    ALTER TABLE audit_runs ADD COLUMN IF NOT EXISTS tier INTEGER DEFAULT 1;
ALTER TABLE audit_runs ADD COLUMN IF NOT EXISTS "filePath" TEXT;
ALTER TABLE audit_runs ADD COLUMN IF NOT EXISTS "parentRunId" UUID REFERENCES audit_runs(id);
  • Service Layer Enhancements:
    • src/server/services/file-prioritizer.ts: The brains behind our weighted scoring logic.
    • src/server/services/repo-audit-service.ts: Orchestrates the discovery, glob filtering, and initial Tier 1 grouping.
    • src/server/services/audit-service.ts: Extended to handle the new repo target type.
  • Workflow Engine Smarts: Our workflow-engine.ts now intelligently parses file ratings from Tier 1 results and dynamically expands into Tier 2 per-file workflows. This dynamic orchestration is critical for scalability.
  • API & UI Integration:
    • The audit-router.ts was updated for repo configuration validation and to trigger the Tier 1 workflow expansion.
    • The Ipcha dashboard now features a dedicated Repo target form, allowing users to select projects, define glob patterns, and set risk thresholds with an intuitive slider.
    • Crucially, the UI also presents audit results in an expandable, human-readable format, clearly distinguishing and linking Tier 1 and Tier 2 runs.
  • Robustness: Extensive edge case tests were implemented to ensure the system behaves predictably under various scenarios.

Lessons Learned and Minor Hurdles

While the "subagent-driven development" approach generally led to a smooth implementation, a few minor snags provided valuable learning opportunities:

  • TypeScript Type Casting: During the Tier 2 expansion in the workflow engine, we encountered a subtle TypeScript issue. A configuration object intended for providerFanOutConfig required a specific as Prisma.InputJsonValue cast, rather than a generic Record<string, unknown>. This highlights the importance of precise type annotations, especially when interacting with ORM-generated types.
  • API Response Structure: When implementing the UI, we discovered that our projects list query returned { items, total, page, limit, hasMore } instead of the expected { projects, total }. This minor API contract mismatch was quickly identified and resolved, emphasizing the need for clear API documentation and thorough testing between frontend and backend.

What's Next?

With the repo-level auditing system now ready for production deployment, our immediate focus shifts to:

  1. Deployment: Pushing the new code, applying the necessary schema migrations, and rebuilding the system.
  2. Validation: Running initial repo audits on /dashboard/ipcha to ensure everything works as expected in a live environment.
  3. Automation: Setting up the AUDIT_CRON_SECRET and configuring hourly cron jobs to ensure continuous, proactive auditing.

Looking further ahead, we're already brainstorming exciting new features, including a "persona rental API" for even more dynamic testing scenarios and deeper integration with knowledge bases like CKB/CognitiveVault for enhanced contextual understanding during audits.

Ipcha's new repo-level auditing system represents a significant step towards truly autonomous and intelligent code quality assurance. By combining smart prioritization with a flexible, tiered workflow, we're empowering developers to build and maintain robust software with greater confidence than ever before. Stay tuned for more updates!