← nyxcore-systems
6 min read

Unveiling the Neural Constellation: A 3D Leap in Vectorized Workflow Insights

We've just shipped 'Neural Constellation,' a groundbreaking 3D visualization that transforms high-dimensional vectorized knowledge into an interactive, intuitive experience. Dive into how we built it, the challenges we overcame, and what it means for understanding complex data workflows.

3DVisualizationReactThreeFiberUMAPVectorDBtRPCKnowledgeGraphAIMachineLearning

Today marks a significant milestone in our journey to make complex data not just manageable, but truly understandable. We're thrilled to announce the successful completion and deployment of Neural Constellation, a powerful new 3D visualization feature designed to bring clarity to our vectorized workflow insights. This isn't just a fancy UI; it's a window into the interconnectedness of our system's memory, offering unprecedented depth for analysis and discovery.

Alongside this visual leap, we've also pushed crucial corrections to our core technical paper, "Ipcha Mistabra," ensuring our documentation accurately reflects the sophisticated mechanisms powering our platform.

Let's dive into the journey of building the Neural Constellation and the insights we gained along the way.

Mapping the Mind: The Neural Constellation

Our goal with Neural Constellation was ambitious: to take high-dimensional vector embeddings, which represent the semantic essence of our system's "memory" (like insights from past workflows, pain points, and solutions), and project them into an intuitive 3D space. This allows users to visually explore relationships, identify clusters of similar information, and uncover hidden patterns that would be impossible to discern from raw data.

Here's a breakdown of the key components that brought this vision to life:

From Vectors to Visuals: The Core Pipeline

The journey begins with raw, high-dimensional vectors, often 1536 dimensions strong. To make these explorable in 3D, we leveraged UMAP (Uniform Manifold Approximation and Projection).

  • Dimensionality Reduction: We implemented src/server/services/umap-projection.ts using umap-js to reduce these complex 1536-dimensional vectors down to an accessible 3D representation. This process preserves the local and global structure of the data, meaning points that are close in high-dimensional space remain close in 3D.
  • Proximity Clustering: Post-projection, we apply proximity clustering to group related "memories" together. This forms the basis of our visual "constellations."
  • Data Fetching with pgvector & tRPC: To feed our visualization, we created a new memory.constellation tRPC endpoint. This endpoint efficiently queries our pgvector database, fetching the vectorized memory data needed for projection and rendering.

Bringing 3D to Life with React Three Fiber

The heart of our 3D experience lies in the powerful React Three Fiber (R3F) library, which allows us to build complex 3D scenes using React components. We integrated a suite of three.js ecosystem libraries:

  • three@0.183.2
  • @react-three/fiber@8.18.0
  • @react-three/drei@9.122.0 (for helpers like controls)
  • @react-three/postprocessing@2.19.1 & postprocessing@6.38.3 (for visual effects like Bloom)

With these tools, we constructed several key visual elements:

  • ParticleField.tsx: This component renders the individual "memory" points as an InstancedMesh. It's highly optimized for performance and includes interactive features like hover states, click handling, and filtering, allowing users to drill down into specific data points.
  • ConstellationFilaments.tsx: To illustrate the relationships between clustered memories, we added ConstellationFilaments.tsx. These lines connect points within the same cluster, visually reinforcing their semantic proximity.
  • PairedArcs.tsx: One of the most insightful features is PairedArcs.tsx. This component draws animated arcs between "pain points" and their corresponding "solutions," providing a clear visual narrative of problem-solving within our workflows.
  • ConstellationHUD.tsx: A crucial part of any complex visualization is context. Our Head-Up Display provides floating statistics, a severity heatmap, a legend, and detailed tooltips to keep users informed without cluttering the 3D space.
  • DetailPanel.tsx: Clicking on a memory point triggers a DetailPanel.tsx, which slides in to present comprehensive information about the selected memory, its associated workflow, and relevant metadata.
  • ConstellationView.tsx: All these components are orchestrated within ConstellationView.tsx, which manages the R3F Canvas, camera controls, and post-processing effects like Bloom to give the constellation a vibrant, ethereal glow.

Finally, we seamlessly integrated the Constellation into our existing memory page, making it a new, interactive tab for users to explore.

Clarifying the Blueprint: Ipcha Mistabra Doc Corrections

While building new features is exciting, maintaining accurate documentation is equally vital. We took the opportunity to address several critical clarifications in our "Ipcha Mistabra" technical paper, ensuring it precisely reflects our current architecture:

  • Disambiguating DA Steps: Corrected Section 4.3 to clarify that the "DA (Decision-Action) step" involves multiple LLM steps configured via providerFanOutConfig, rather than a single review step. This is a subtle but important distinction in how our system processes decisions.
  • Beyond Regex: Fixed Section 4.5, emphasizing that Ipcha utilizes a sophisticated provider/lens fan-out mechanism for processing, moving beyond simple regex section splitting. This highlights the advanced capabilities of our system.
  • The 5-Step Pipeline: Documented the actual, refined 5-step pipeline: Prepare → Adversarial Analysis → Synthesis → Arbitration → Results. This provides a clear and accurate overview of our core process.
  • Ethical Considerations: Added an {{ethics}} template variable and insightScope: "ethic" persistence, formalizing how we track and integrate ethical considerations throughout our workflows.

These corrections ensure that our technical paper remains a true and reliable reference for understanding our platform's underlying intelligence.

Lessons Learned: Navigating the Development Terrain

No significant feature ships without its share of challenges. Here's what we encountered and how we navigated them:

  • React Three Fiber Versioning: Our initial attempt was to jump to the latest v9 of @react-three/fiber. However, this version has a peer dependency on React 19, while our project is currently on React 18.3.1.
    • Solution: We pragmatically pinned @react-three/fiber to v8.18.0, the last version compatible with React 18. All other peer dependencies resolved cleanly, allowing us to proceed without a major React upgrade. This highlights the importance of checking peer dependencies early in a project!
  • ESLint and Build-Time Code Hygiene: As the codebase grew, we hit some common ESLint errors during the production build:
    • An unused colors variable in ParticleField.tsx.
    • An unused THREE import in ConstellationFilaments.tsx.
    • points and clusters arrays not wrapped in useMemo in ConstellationView.tsx, leading to unnecessary re-renders.
    • Solution: Simple but crucial fixes: removed the unused colors variable and useMemo import, deleted the unnecessary THREE import, and wrapped the points and clusters data structures in useMemo to optimize performance and prevent redundant calculations. This reinforced the value of strict linting and performance awareness.
  • tRPC Context Nuance: A minor but important detail was catching that our tRPC context exposes ctx.tenantId directly, rather than ctx.tenant.id. This was identified and fixed during implementation, preventing potential data access issues.

What's Next? The Horizon of Discovery

With Neural Constellation now live on main and deploying to production, our immediate next steps involve:

  1. Production Verification: Ensuring the deploy completes successfully and the feature is stable in a live environment.
  2. Live Testing: Thoroughly testing the Constellation at https://nyxcore.cloud/dashboard/memory to confirm its functionality and performance.
  3. Data Generation: Running workflows with review steps to generate the necessary embedding data, populating the Constellation with rich, insightful particles.
  4. Admin Configuration: Guiding users to set their fallback provider in Admin > LLM Defaults for optimal performance.
  5. Documentation Pipeline: Re-running our docs pipeline to publish the 13 pending documentation updates.

Looking further ahead, we're already envisioning exciting enhancements, such as:

  • Camera Fly-in on Click: A more immersive experience when selecting a memory point.
  • Embedding-Generated Search: Moving beyond title-only search to allow semantic search within the Constellation.
  • Mobile Responsiveness: Adapting the 3D experience for on-the-go exploration.

This project has been a fantastic journey, pushing the boundaries of what's possible in visualizing complex, vectorized knowledge. We're incredibly excited about the insights Neural Constellation will unlock for our users and the future of understanding our system's memory. Stay tuned for more updates!