Conquering Code Intelligence: Our Journey Integrating CKB into Production

It's 3 PM on a Thursday, and the "Letter to Myself" for this development session feels particularly satisfying. We've officially pushed CKB (Code Knowledge Base), our new code intelligence engine, to production! This wasn't just a small feature; it was a full, end-to-end integration, from spinning up new Docker containers to a fully interactive UI, now live and providing insights into our codebase.

The "Code Intelligence" tab is operational, giving us a real-time pulse on code architecture, hotspots, and potential dead code. Even as I write this, the background agents are busy generating detailed technical and executive summaries of the integration itself. It's a fantastic feeling to see such a complex system come to life.

The Mission: Bringing Code Intelligence to Life

Our goal was ambitious: integrate CKB, a powerful code analysis tool, deeply into our platform. This meant not just running it, but making it a seamless part of our project workflows, accessible through our UI, and robust enough for production. We broke it down into two main phases:

Phase 1: The Core Integration – Laying the Foundation

This phase was all about the plumbing. We needed CKB to run reliably, store its data, and be callable by our application.

Dockerization: We containerized CKB, building ghcr.io/simplyliz/ckb:latest from our CodeMCP repository and deploying it as a dedicated service on our production server. This ensured isolation and consistent environments.
Data Persistence with Prisma: We introduced a new ProjectCkbIndex model in Prisma, complete with Row-Level Security (RLS) policies and back-relations, to store CKB's analysis outputs securely and efficiently.
Service Layer: Our src/server/services/ckb-client.ts became the orchestrator, a wrapper around docker exec to invoke CKB's 13 distinct analysis functions.
tRPC Router: We exposed these analysis capabilities via src/server/trpc/routers/ckb.ts, providing 13 tRPC procedures. Crucially, we implemented async fire-and-forget indexing to avoid blocking the UI during long analysis tasks.
Workflow Engine Integration: We even wired {{ckb}} into our workflow engine via src/server/services/ckb-content-loader.ts, allowing for dynamic, context-aware content generation based on CKB data.
Automatic Indexing: A small but critical piece: CKB analysis now automatically triggers whenever a repository link is added or updated in src/server/trpc/routers/projects.ts.
Testing: We ensured robustness with 17 passing tests – 10 for the client and 7 for the content loader.

Phase 2: Code Intelligence UI – Bringing Insights to the Frontend

With the backend humming, it was time to expose CKB's power to our users.

Dedicated UI Tab: src/components/projects/code-intelligence-tab.tsx became the home for CKB's output. It features an overview section with key metrics and detailed breakdowns for different analysis types.
Sidebar Integration: We added "Code Intel" to the "Development" group in our project page sidebar, making it easily discoverable.
Progress Bar: Long-running analysis tasks needed feedback. We implemented a 5-step progress bar (Clone → Architecture → Hotspots → Audit → Dead Code) to keep users informed.
Real-time Polling: Using useEffect and useState, the UI polls the backend every 2 seconds during processing, giving a near real-time view of the analysis status.

Lessons from the Trenches: The "Pain Log" Turned Wisdom

No integration of this scale goes without its bumps. The real learning often happens when things don't work as expected. Here are some of the critical fixes and lessons we learned along the way:

1. The Elusive Docker CLI & Workdir Dance

The Problem: Our initial attempts to run CKB commands from our application container against the CKB container failed with unknown command "architecture" and --repo flag errors.
The Deeper Issue: CKB's CLI command for architecture analysis was arch, not architecture, and it expected the target repository path to be set as the working directory (-w) for docker exec, not passed via a --repo flag which didn't exist.
The Fix & Lesson:
bash
```
# Failed attempt:
# docker exec nyxcore-ckb-1 ckb architecture --repo /data/repos/xxx --format json

# The working solution:
docker exec -w /data/repos/xxx nyxcore-ckb-1 ckb arch --format json
```
Lesson Learned: Always verify CLI syntax, command names, and expected flags directly from the tool's documentation or help output, especially when dealing with new versions or unique container setups. Don't assume. Also, understand how docker exec -w functions for setting context within the target container.

2. Docker Socket Access & Permissions Hell

The Problem: Our app container couldn't execute docker exec commands. First, docker: executable not found in $PATH, then permission denied on /var/run/docker.sock.
The Deeper Issue: The app container didn't have the docker-cli installed by default, and even after installing it, it lacked the necessary permissions to communicate with the host's Docker daemon via the socket.
The Fix & Lesson:
dockerfile
```
# In app Dockerfile:
RUN apt-get update && apt-get install -y docker-cli

# In docker-compose or Kubernetes deployment:
volumes:
  - /var/run/docker.sock:/var/run/docker.sock:ro
group_add:
  - "988" # Assuming 'docker' group GID on the host is 988
```
Lesson Learned: When integrating Docker containers that need to control other Docker containers, meticulously plan Docker socket access (read-only where possible), ensure the docker-cli is installed in the calling container, and correctly map host group IDs (like the docker group) to the container to grant necessary permissions.

3. Private Repo Cloning: Token Fallback Strategy

The Problem: CKB failed to clone private GitHub repositories. Our resolveGitHubToken() function wasn't providing the correct token.
The Deeper Issue: The function was designed to query github_tokens (personal user tokens), but for project-level private repos, we needed to use api_keys which hold tenant-specific GitHub tokens.
The Fix & Lesson: We implemented a fallback mechanism: try to resolve from github_tokens first, and if that fails, try api_keys with provider: "github". Lesson Learned: Design robust authentication resolution logic. Anticipate different token sources (e.g., personal user tokens vs. organization/tenant-wide API keys) and implement clear, secure fallbacks for accessing private resources.

4. Navigating CKB's Evolving Data Structures

The Problem: We were trying to render CKB data with assumptions like h.risk.toFixed(1) and spreading raw arrays [...hotspots]. This led to runtime errors because the data wasn't shaped as expected.
The Deeper Issue: CKB's output schema evolved. For instance, it returned { hotspots: [{filePath, score, riskLevel}] } instead of [{file, risk}]. Field names changed (filePath not file, score not risk, items not findings, riskLevel not severity). Also, sometimes CKB returns wrapped objects where we expected raw arrays.

The Fix & Lesson:

typescript

// Example defensive handling:
type CKBHotspot = { filePath: string; score: number; riskLevel: string };
type CKBResponse = { hotspots?: CKBHotspot[] };

const processHotspots = (data: Record<string, unknown>): CKBHotspot[] => {
  if (data && typeof data === 'object' && 'hotspots' in data) {
    const rawHotspots = (data as CKBResponse).hotspots;
    if (Array.isArray(rawHotspots)) {
      return rawHotspots.map(h => ({
        filePath: h.filePath || 'unknown',
        score: Number(h.score) || 0, // Ensure numeric safety
        riskLevel: h.riskLevel || 'UNKNOWN'
      }));
    }
  }
  return [];
};

Lesson Learned: Always validate external API/tool data structures rigorously. Implement defensive coding with Array.isArray() checks, typeof guards, and robust type assertions, especially when dealing with dynamic or evolving external data sources. Default to safe values (Number(h.score) || 0) to prevent runtime errors.

5. Conditional Polling with `refetchInterval`

The Problem: We tried to use refetchInterval: (query) => { ... } callback syntax for conditional polling based on the query status (e.g., only poll if isLoading or isFetching). This resulted in a cryptic "(intermediate value) is not iterable" error.
The Deeper Issue: This likely pointed to a subtle tRPC/React Query version incompatibility or an unexpected behavior with the callback syntax in our specific setup.
The Fix & Lesson: We simplified it using a useState hook to control polling directly.
typescript
```
const [isPolling, setIsPolling] = useState(false);

// ... inside useEffect for analysis status ...
if (status === 'PROCESSING') {
  setIsPolling(true);
} else {
  setIsPolling(false);
}

// ... in useQuery options ...
refetchInterval: isPolling ? 2000 : false,
```
Lesson Learned: While advanced refetchInterval callbacks are powerful, be wary of subtle version incompatibilities or unexpected API behaviors in UI libraries (tRPC/React Query). Sometimes, simpler state-driven toggles (useState + useEffect) are more robust and easier to debug than complex callback logic, especially when dealing with conditional fetching.

Current State and What's Next

The system is live!

Our production server (root@46.225.232.35) is running the latest commit 8af3329.
The CKB container (nyxcore-ckb-1) is running CKB v8.1.0, built from our CodeMCP repo.
The project_ckb_indexes table is populated (currently with data from the Clarait-Auth project), protected by RLS.
Redis is caching CKB summaries with a 1-hour TTL, and Anthropic API credits are restored, so embeddings are working again.
The ckb-docs agent is diligently generating docs/ckb-integration-technical.md and docs/ckb-integration-executive-summary.md in the background.

Immediate next steps involve waiting for those docs to finish, committing them, and then diving into Phase 3: Webhook endpoint for auto-reindex on GitHub push. We'll also be refining our content-loader tests and considering SCIP index support for even richer symbol analysis.

This integration was a significant undertaking, but seeing the Code Intelligence tab light up with actionable insights makes every "pain log" entry feel worth it. Here's to more intelligent code!

json

{
  "thingsDone": [
    "CKB container deployed to production",
    "Prisma model for CKB indexes with RLS implemented",
    "Service layer for CKB analysis functions (`docker exec` wrapper)",
    "tRPC router with 13 procedures for CKB, including async indexing",
    "CKB content loader integrated into workflow engine",
    "Automatic CKB indexing on repository link updates",
    "Full Code Intelligence UI tab with overview cards and detail sections",
    "Sidebar navigation for Code Intel tab",
    "5-step progress bar for CKB analysis",
    "Real-time polling for analysis status in UI",
    "Robust error handling and defensive coding for CKB data"
  ],
  "pains": [
    "Incorrect CKB CLI command names and flag usage (`architecture` vs `arch`, `--repo` vs `-w`)",
    "Docker socket access and permissions issues for `docker exec` (missing `docker-cli`, `permission denied`)",
    "GitHub token resolution for private repositories (personal vs. tenant tokens)",
    "Mismatched CKB data structures and field names (e.g., `file` vs `filePath`, `risk` vs `score`)",
    "Unexpected behavior with `refetchInterval` callback in tRPC/React Query"
  ],
  "successes": [
    "Successful end-to-end CKB integration on production",
    "Overcoming complex Docker networking and permission challenges",
    "Implementing a flexible GitHub token fallback mechanism",
    "Developing a robust and user-friendly Code Intelligence UI",
    "Establishing a stable foundation for future CKB enhancements"
  ],
  "techStack": [
    "CKB (Code Knowledge Base)",
    "Docker",
    "Prisma",
    "PostgreSQL",
    "tRPC",
    "React",
    "Next.js",
    "TypeScript",
    "Redis",
    "GitHub Actions (for CI/CD)",
    "Anthropic API (for embeddings)"
  ]
}