From Fantasyland to Filesystem: Grounding LLMs with Real Code Context Using `{{claudemd}}` and `{{fileTree}}`

In the exciting world of AI-powered development, Large Language Models (LLMs) are revolutionary. They can generate code, explain complex concepts, and even refactor entire modules. Yet, ask an LLM to work on a specific codebase it hasn't been explicitly trained on, and you'll often encounter a frustrating phenomenon: hallucination.

One of the most common and disruptive forms of hallucination we've faced is when an LLM invents file paths, directory structures, or even entire architectural patterns that simply don't exist in the actual repository. This leads to wasted time, incorrect suggestions, and a significant drop in trust in the AI's capabilities.

Our goal was clear: eliminate LLM hallucinations related to file paths and project architecture by grounding prompts in the real codebase context. Today, I'm excited to share how we achieved a significant leap forward by introducing two powerful new template variables into our workflow engine: {{claudemd}} and {{fileTree}}.

The Problem: LLMs Dreaming Up Codebases

Imagine asking an LLM to "add a new feature to src/components/dashboard/analytics.tsx" only for it to suggest creating app/features/analytics/DashboardAnalytics.jsx. Or perhaps it references a utils/helpers.ts file that was refactored out months ago. This "architectural amnesia" stems from the fact that while LLMs are incredibly knowledgeable, they lack real-time, precise context about your specific project's current state. They rely on their training data, which might be outdated or too generic.

This isn't just an annoyance; it's a productivity killer. Developers spend time correcting non-existent paths, debugging against phantom files, and ultimately losing faith in the AI's utility.

Our Solution: Injecting Reality with {{claudemd}} and {{fileTree}}

To combat this, we decided to explicitly provide the LLM with the context it desperately needed. We implemented a system to fetch and inject two critical pieces of information directly into our prompts:

1. {{claudemd}}: A placeholder for project-specific documentation, typically from a CLAUDE.md (or README.md) file. This allows teams to provide high-level context, design decisions, specific instructions, or even a mini-manifesto for how the AI should approach the codebase.
2. {{fileTree}}: A markdown-formatted representation of the actual repository's directory structure. This is the ultimate truth-teller, showing the LLM exactly what files and folders exist, and where.

Let's dive into how we brought this to life.

Under the Hood: Building the Contextual Bridge

Our recent development session focused entirely on integrating these variables into our existing workflow engine. Here’s a breakdown of the key steps:

1. Fetching the Source of Truth: The Repository File Tree

To get an accurate, up-to-date file tree, we leveraged the GitHub Git Trees API (/git/trees/{branch}?recursive=1). This powerful endpoint allows us to fetch a full directory listing of a repository in a single, efficient API call.

We implemented a new fetchRepoTree() function in src/server/services/github-connector.ts. Crucially, this function also includes:

• Noise Filtering: We filter out common development artifacts like node_modules, .git, .next, dist, and lock files, ensuring the LLM only sees relevant code structure.
• Branch Fallback: To maximize compatibility, fetchRepoTree gracefully falls back from the main branch to master if main isn't found, accommodating older or differently configured repositories.
• Entry Cap: The file tree is capped at 500 entries to prevent excessively large prompts, which could hit token limits or degrade performance.

The output is then formatted as a clean markdown code block, ready for prompt injection.

2. Loading Project-Specific Documentation (CLAUDE.md)

Alongside the file tree, we introduced loadClaudeMdContent() in src/server/services/workflow-engine.ts. This function attempts to load content from a CLAUDE.md file at the root of each linked repository. If CLAUDE.md isn't found, it intelligently falls back to README.md, ensuring that some form of project context is almost always available.

3. Seamless Integration into the Workflow Engine

To make these new contextual variables available to our LLM prompts, we:

• Extended ChainContext: Added claudeMdContent and fileTreeContent fields to our ChainContext object, which carries all relevant data throughout a workflow execution.
• Resolved Variables in resolvePrompt(): Our resolvePrompt() function, responsible for injecting dynamic data into prompt templates, was updated to recognize and substitute {{claudemd}} and {{fileTree}} with their fetched content.
• Parallel Loading for Performance: Both loadClaudeMdContent() and loadFileTreeContent() now run in parallel via Promise.all during workflow startup. This ensures minimal latency overhead, as they load alongside existing document and consolidation steps.

4. Targeted Prompt Engineering for Maximum Impact

The true power of these variables comes from their strategic placement within our prompt templates. We updated five critical prompt templates in src/lib/constants.ts that were identified as primary sources of hallucination:

• extensionAnalyze: The initial analysis step, now grounded in real structure from the start.
• extensionPrompt: Our main implementation prompts, where hallucinated paths were most common.
• secRecon: For security reconnaissance, ensuring the LLM understands the actual attack surface.
• secPrompts: Security fix prompts, guiding the LLM to existing files.
• deepPrompt: For deep build implementation, preventing the creation of non-existent modules.

Beyond just injecting the variables, we also updated the system prompts for all these templates with a crucial directive:

"MUST reference real file paths from the provided file tree — never invent or guess paths."

This explicit instruction reinforces the importance of using the provided context, guiding the LLM towards more accurate and actionable responses.

Lessons Learned & Smooth Sailing

This particular session was remarkably smooth. The feature addition was straightforward, benefiting from well-defined APIs and our existing modular architecture. This stands in contrast to some previous sessions where we grappled with complex authentication flows, tricky Server-Sent Events (SSE) implementations, or subtle Prisma ORM gotchas. A clean session like this is a testament to the value of good upfront design and robust tooling.

One small but important consideration, however, is that the GitHub Trees API requires repo-level read access on the Personal Access Token (PAT) stored via our Bring Your Own Key (BYOK) system. This is a crucial security and configuration point for anyone looking to implement a similar solution.

What's Next?

With the core implementation complete, our immediate next steps involve:

1. End-to-End Testing: Running our Extension Builder workflow with linked repositories to verify that prompts consistently contain real file paths and CLAUDE.md content.
2. Token Budget Verification: Ensuring that the added file tree and CLAUDE.md content don't inadvertently blow the LLM's context window, especially for larger repositories. The 500-file cap is a good start, but real-world testing is key.
3. Expanded Usage: If testing reveals further hallucination in downstream steps, we'll consider adding {{claudemd}} and {{fileTree}} to more templates (e.g., extensionFeatures, extensionExtend, secRemediation).
4. Documentation: Updating our internal and external documentation to list {{claudemd}} and {{fileTree}} as supported and highly recommended template variables.

Conclusion

By introducing {{claudemd}} and {{fileTree}}, we've taken a significant step towards creating a more reliable and less frustrating experience with AI-powered code generation. We're moving from a world where LLMs dream up their own architectural fantasylands to one where they are firmly grounded in the reality of your actual codebase. This dramatically improves the accuracy, utility, and trustworthiness of our AI assistant, making it a truly indispensable tool for developers.

Stay tuned for more updates as we continue to refine and expand our AI capabilities!