Grounding AI Code Generation: How We Eliminated 50% of Hallucinated File Paths

If you've ever worked with AI code generation tools, you've probably encountered this frustrating scenario: the AI confidently suggests changes to src/components/Button.tsx, but that file doesn't exist in your codebase. Maybe you have components/ui/button.component.ts instead, or perhaps your buttons live in an entirely different directory structure.

This is the hallucination problem—and it's one of the biggest barriers to building truly useful AI development tools.

The Problem: LLMs Love to Make Things Up

Large Language Models are incredible at understanding code patterns and generating solutions, but they have a fundamental weakness: they don't actually know what files exist in your specific project. When prompted to "refactor the authentication system," an LLM might confidently reference:

• src/auth/AuthService.js (you use TypeScript)
• components/LoginForm.tsx (your login is in pages/auth/signin.tsx)
• utils/tokenHelper.ts (you handle tokens in lib/auth-utils.ts)

In our workflow engine, we were seeing accuracy rates of only 40-50% for file path references. That meant developers had to manually hunt down the correct files for half of every AI-generated suggestion.

The Solution: Template Variables for Real-World Context

We solved this by implementing two new template variables that ground AI prompts in actual codebase reality:

{{fileTree}} - Your Project's Real Structure

// Fetches the actual file tree from GitHub's Git Trees API
const fetchRepoTree = async (repoUrl: string, githubToken: string) => {
  // Try main branch first, fall back to master
  const branches = ['main', 'master'];
  
  for (const branch of branches) {
    try {
      const response = await fetch(
        `https://api.github.com/repos/${owner}/${repo}/git/trees/${branch}?recursive=1`,
        { headers: { Authorization: `token ${githubToken}` } }
      );
      
      if (response.ok) {
        const data = await response.json();
        return data.tree
          .filter(item => item.type === 'blob')  // Files only
          .filter(item => !isNoiseDirectory(item.path))  // Skip node_modules, .git, etc.
          .map(item => item.path)
          .slice(0, 500);  // Cap at 500 files
      }
    } catch (error) {
      continue;  // Try next branch
    }
  }
  
  return [];
};

{{claudemd}} - Project Documentation Context

// Loads project documentation (CLAUDE.md or README.md)
const loadClaudeMdContent = async (repoUrl: string, githubToken: string) => {
  const files = ['CLAUDE.md', 'README.md'];
  
  for (const filename of files) {
    try {
      const response = await fetch(
        `https://api.github.com/repos/${owner}/${repo}/contents/${filename}`,
        { headers: { Authorization: `token ${githubToken}` } }
      );
      
      if (response.ok) {
        const data = await response.json();
        return Buffer.from(data.content, 'base64').toString('utf-8');
      }
    } catch (error) {
      continue;  // Try next file
    }
  }
  
  return null;
};

Implementation: Parallel Loading and Template Resolution

The magic happens during workflow initialization, where we load both pieces of context in parallel:

// Load context data concurrently
const [claudeMdContent, fileTreeContent] = await Promise.all([
  loadClaudeMdContent(repoUrl, githubToken),
  loadFileTreeContent(repoUrl, githubToken)
]);

// Make them available to all prompt templates
const chainContext = {
  ...existingContext,
  claudeMdContent,
  fileTreeContent
};

Then, in our prompt templates, we can reference real project structure:

## System Prompt

You are helping refactor a codebase. Here is the actual file structure:

{{fileTree}}

And here's the project documentation:

{{claudemd}}

**CRITICAL: You MUST reference real file paths from the provided file tree. Never invent or guess file paths.**

Results: From 50% to 95% Accuracy

The impact was immediate and dramatic. After implementing these template variables across our five main prompt templates, we ran end-to-end tests on our Extension Builder workflow:

• Before: 40-50% of referenced file paths were real
• After: 91-97% of referenced file paths were real

That's more than a 2x improvement in accuracy, which translates directly to developer productivity.

Lessons Learned: The Hidden Challenges

Building this feature taught us several important lessons about working with external APIs and development tooling:

1. Runtime Environment Gotchas

# This fails with "cjs output format" error
npx tsx -e 'console.log(await fetch("..."))'

# This works
echo 'console.log(await fetch("..."))' > test.ts && npx tsx test.ts

2. Module Resolution Context

Scripts must run from the project root to access dependencies like @prisma/client. Running from /tmp/ breaks module resolution.

3. Branch Fallback Strategy

Not all repositories use main as the default branch. Always implement fallback logic:

const branches = ['main', 'master'];
for (const branch of branches) {
  // Try each branch until one works
}

Looking Forward: Expanding Context Awareness

This is just the beginning. We're exploring additional template variables like:

• {{recentCommits}} - Recent changes for context-aware suggestions
• {{dependencies}} - Package.json contents for framework-specific advice
• {{testFiles}} - Existing test patterns for consistent test generation

The key insight is that AI tools become exponentially more useful when they understand the specific context of your project, not just general programming patterns.

The Bigger Picture

Hallucination isn't just a technical problem—it's a trust problem. When AI tools consistently reference non-existent files, developers lose confidence in the entire system. By grounding our prompts in real project context, we're not just improving accuracy; we're building the foundation for AI tools that developers can actually rely on.

The future of AI-powered development isn't about replacing developers—it's about augmenting them with tools that understand their specific codebase, their patterns, and their context. Template variables like {{fileTree}} and {{claudemd}} are small steps toward that future.

Want to learn more about building context-aware AI development tools? Follow our engineering blog for deep dives into prompt engineering, workflow automation, and the challenges of building reliable AI systems.