{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"A complete technical guide to building a production-ready RAG-powered real-time voice assistant — covering RTVI architecture, Google Gemini embeddings, MongoDB Atlas vector search, FastAPI equipment/stream endpoints, document chunking, and full AWS ECS/Fargate deployment."}]}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"Table of Contents"}]},{"type":"orderedList","attrs":{"start":1,"type":null},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#1-project-architecture-overview","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"Project Architecture Overview"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#2-core-concepts-client-person--pipeline","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"Core Concepts: Client, Person & Pipeline"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#3-rtvi-architecture-transport--processor--observer","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"RTVI Architecture: Transport → Processor → Observer"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#4-pipeline-components--class-hierarchy","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"Pipeline Components & Class Hierarchy"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#5-five-step-pipeline-task-flow","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"Five-Step Pipeline Task Flow"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#6-google-gemini-embeddings--mongodb-atlas-vector-store","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"Google Gemini Embeddings & MongoDB Atlas Vector Store"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#7-fastapi-equipment-routes-vs-stream-endpoints","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"FastAPI: Equipment Routes vs Stream Endpoints"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#8-data-chunking--indexing-schema","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"Data Chunking & Indexing Schema"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#9-project-structure--core-code-walkthrough","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"Project Structure & Core Code Walkthrough"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#10-environment-setup--api-key-configuration","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"Environment Setup & API Key Configuration"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#11-mongodb-atlas-vector-store-setup","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"MongoDB Atlas Vector Store Setup"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#12-full-implementation-from-clone-to-running-locally","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"Full Implementation: From Clone to Running Locally"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#13-cicd-pipeline-github-actions--ecr--ecsfargate","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"CI/CD Pipeline: GitHub Actions → ECR → ECS/Fargate"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"link","attrs":{"href":"https://claude.ai/chat/f47b220c-bde6-408a-b7c7-cc2c9888f896#14-real-time-call-center-use-case","target":"_blank","rel":"noopener noreferrer nofollow","class":"text-blue-600 underline"}}],"text":"Real-Time Call Center Use Case"}]}]}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"1. Project Architecture Overview"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Goal"}]},{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Build an end-to-end "},{"type":"text","marks":[{"type":"bold"}],"text":"RAG-powered real-time voice assistant"},{"type":"text","text":" that:"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Captures voice input via a browser-based frontend"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Transcribes speech using Deepgram (STT)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Retrieves relevant knowledge from MongoDB Atlas (vector search)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Generates grounded responses via Grok LLM"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Synthesizes speech via ElevenLabs (TTS)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Streams audio back to the user in real time"}]}]}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Full Technology Stack"}]},{"type":"table","content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Category"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Tool / Service"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Primary Role"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Backend Framework"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"FastAPI"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"API endpoints, async handling"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Vector Store"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"MongoDB Atlas"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Store embeddings for similarity search"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Embedding Provider"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Google Gemini"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Generate 768-dim vector representations"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"LLM"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Grok"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Generate responses and function calls"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Speech-to-Text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Deepgram, Cartesia"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Convert voice to text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Text-to-Speech"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"ElevenLabs (11Labs), Cartesia"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Convert text to spoken output"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Orchestration"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"PipeCat"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Wire STT, LLM, and TTS into a pipeline"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Frontend"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"React, Tailwind, WaveSocket"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"UI and real-time audio communication"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Containerization"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Docker"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Build and run services consistently"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"CI/CD"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"GitHub Actions"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Automated testing and deployment"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Cloud Platform"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"AWS (ECR, ECS/Fargate, ALB, CloudWatch, Secrets Manager)"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Hosting, scaling, monitoring"}]}]}]}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Data Flow: End-to-End"}]},{"type":"codeBlock","attrs":{"language":"javascript"},"content":[{"type":"text","text":"[User speaks in browser]\n │\n ▼\n[WaveSocket / WaveRTC] ──── audio frames (≈0.1s each) ────▶\n │\n ▼\n[Transport Input Layer]\n │\n ▼\n[Deepgram STT] ──── transcribed text ────▶\n │\n ▼\n[Grok LLM] ──── generates function calls ────▶\n │\n ▼\n[MongoDB Atlas Vector Store] ──── similarity search ────▶\n (embedded via Google Gemini)\n │\n retrieved documents\n │\n ▼\n[Grok LLM] ──── generates grounded answer ────▶\n │\n ▼\n[ElevenLabs TTS] ──── synthesized audio ────▶\n │\n ▼\n[WaveSocket] ──── streams audio back to browser ────▶\n │\n ▼\n[User hears the response]\n"}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"2. Core Concepts: Client, Person & Pipeline"}]},{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Before diving into code, three foundational definitions anchor the architecture:"}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Definition — Client:"},{"type":"text","text":" The front-end endpoint where a user interacts with the system (e.g., web app, mobile app, Zoom call). The client captures audio and renders responses."}]}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Definition — Person:"},{"type":"text","text":" The human user who speaks to the system and receives spoken answers."}]}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Definition — Pipeline:"},{"type":"text","text":" A deterministic sequence of processing steps — analogous to an assembly line — that transforms raw audio input into a final spoken response."}]}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"How They Interact"}]},{"type":"codeBlock","attrs":{"language":"javascript"},"content":[{"type":"text","text":"[Person] ──speaks──▶ [Client (React/WaveSocket)] ──audio frames──▶ [Pipeline]\n │\n ◀──spoken answer── [Client] ◀──synthesized audio── [Pipeline output] │\n"}]},{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"The client and person interact bidirectionally through the transport layer, while the pipeline handles all internal processing."}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"3. RTVI Architecture: Transport → Processor → Observer"}]},{"type":"customImage","attrs":{"src":"https://67dc5f63b8b6a6f0ea39b99a.storage.fermion.app/public-cached/markdown-editor-file/file-unique-id/ijyybwnsrfwl4cc4my6cxm2uknsysakah6w8b6earh-upofllsnlqrrbp5g2athuj0u825geigsg_haadl_u3lbhns9fhuwtxiqz","alt":null,"width":600,"height":327}},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"The Three Layers"}]},{"type":"table","content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Layer"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Responsibility"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Transport"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Streams audio frames (WebSocket / WaveRTC) between client and backend. Abstracts network protocols."}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Processor"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Executes the five-step pipeline (STT → embedding → vector retrieval → LLM → TTS)"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Observer"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Monitors real-time events (user_start, user_stop, ai_done) — emits green ✅ for success, red ❌ for failure. Acts like a traffic controller."}]}]}]}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Definition — Transport:"},{"type":"text","text":" The I/O conduit (e.g., WaveSocket, WaveRTC) that streams audio chunks (frames) from the client to the backend and returns the generated audio. It abstracts the networking layer so the pipeline logic is decoupled from protocol details."}]}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Definition — Frame:"},{"type":"text","text":" The smallest unit of audio data exchanged between client and backend — approximately 0.1 seconds of audio. Analogous to a single frame in a video stream. Sending small frames minimizes latency."}]}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Observer Event Model"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"type":"text","text":"# Observer monitors three key events in the pipeline lifecycle\n\nclass PipelineObserver:\n \n def on_user_start(self):\n \"\"\"Fired when VAD detects the user has started speaking.\"\"\"\n logger.info(\"✅ User started speaking — activating STT\")\n \n def on_user_stop(self):\n \"\"\"Fired when VAD detects silence — user has stopped speaking.\"\"\"\n logger.info(\"✅ User stopped — triggering LLM inference\")\n \n def on_ai_done(self):\n \"\"\"Fired when TTS audio has finished playing back.\"\"\"\n logger.info(\"✅ AI response complete — ready for next turn\")\n \n def on_error(self, component: str, error: Exception):\n \"\"\"Fired on any pipeline failure — isolates which step broke.\"\"\"\n logger.error(f\"❌ Error in {component}: {error}\")\n # Architect can intervene without diving into every code file\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Transport: WaveRTC vs WebSocket"}]},{"type":"table","content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Protocol"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Best For"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Latency"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"WaveRTC"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Real-time audio/video; peer-to-peer"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Ultra-low (<50ms)"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"WebSocket"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Bidirectional text/binary streaming"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Low (<100ms)"}]}]}]}]},{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"WaveRTC is preferred for the voice pipeline because it is specifically optimized for audio transmission with minimal buffering."}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"4. Pipeline Components & Class Hierarchy"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Core Classes"}]},{"type":"table","content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Class"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Purpose"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Pipeline"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Container that defines the overall workflow (transport ↔ processor ↔ observer)"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"PipelineTask"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Concrete step inside the pipeline (e.g., STT, LLM Inference, TTS)"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Transport"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Implements input and output methods; encapsulates networking details"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Runner"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Execution engine that boots the pipeline, monitors health, and drives the event loop"}]}]}]}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Pipeline Class Skeleton"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"type":"text","text":"# Conceptual class hierarchy — bot.py implementation pattern\n\nfrom pipecat.pipeline.pipeline import Pipeline\nfrom pipecat.pipeline.runner import PipelineRunner\nfrom pipecat.pipeline.task import PipelineTask, PipelineParams\nfrom pipecat.transports.wavesocket import WaveSocketTransport\n\nclass MyRAGPipeline(Pipeline):\n \"\"\"\n The Pipeline container wires all tasks together.\n Runner invokes MyRAGPipeline().run() — analogous to a CPU\n executing an instruction stream.\n \"\"\"\n \n def __init__(self):\n super().__init__()\n # Transport: handles audio I/O with the client\n self.transport = WaveSocketTransport(\n host=\"0.0.0.0\",\n port=8765\n )\n # Processing tasks\n self.task_stt = SpeechToTextTask() # Deepgram\n self.task_rag = RAGRetrievalTask() # Gemini + MongoDB Atlas\n self.task_llm = LLMInferenceTask() # Grok\n self.task_tts = TextToSpeechTask() # ElevenLabs\n \n def run(self):\n self.transport.start()\n self.process()\n\n def define_functions(self):\n \"\"\"Each task is represented by a method (function-to-function flow).\"\"\"\n pass\n\n def register_event_handlers(self):\n \"\"\"Register callbacks for on_start, on_stop, and on_error events.\"\"\"\n self.on(\"user_start\", self.observer.on_user_start)\n self.on(\"user_stop\", self.observer.on_user_stop)\n self.on(\"ai_done\", self.observer.on_ai_done)\n\n\n# Entry point — Runner boots the pipeline\nasync def main():\n pipeline = MyRAGPipeline()\n runner = PipelineRunner()\n await runner.run(PipelineTask(pipeline, PipelineParams(\n allow_interruptions=True,\n enable_metrics=True\n )))\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Runner Execution Flow"}]},{"type":"codeBlock","attrs":{"language":"javascript"},"content":[{"type":"text","text":"1. [Initialize Runner]\n │ Load pipeline config, resolve dependencies, start transport\n ▼\n2. [Event Loop — ≈10Hz / 10 cycles per second]\n │ Continuously listen for inbound audio frames\n ▼\n3. [Observer monitors events]\n ├── user_start → activate STT\n ├── user_stop → trigger LLM + TTS\n └── ai_done → ready for next turn\n ▼\n4. [Performance tuning]\n ├── Latency: minimize via small frames + WaveRTC\n └── Scalability: tune batch size / concurrency in transport params\n"}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"5. Five-Step Pipeline Task Flow"}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Definition — Pipeline:"},{"type":"text","text":" A deterministic sequence of processing stages that transforms raw audio into a final spoken response."}]}]},{"type":"codeBlock","attrs":{"language":"javascript"},"content":[{"type":"text","text":"┌─────────────────────────────────────────────────────────────────┐\n│ FIVE-STEP PIPELINE │\n│ │\n│ Step 1 Step 2 Step 3 Step 4 Step 5 │\n│ │\n│ [Listen]──▶[Speech-to- ──▶[Under- ──▶[Reply ──▶[Text- │\n│ Text] standing] Generate] to- │\n│ Speech] │\n│ │\n│ Capture Deepgram or Grok, Gemini LLM answer 11Labs or │\n│ ≈0.1s Cartesia GPT + RAG Cartesia │\n│ frames STT NLU retrieval TTS │\n└─────────────────────────────────────────────────────────────────┘\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Step-by-Step Detail"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"type":"text","text":"# Illustrative pipeline execution — bot.py core logic\n\nasync def process_turn(audio_frames: list[bytes]) -> bytes:\n \n # ── Step 1: Listen ────────────────────────────────────────────\n # Audio frames arrive from WaveSocket transport (≈0.1s each)\n # VAD has already confirmed these are speech frames\n \n # ── Step 2: Speech-to-Text ────────────────────────────────────\n transcription = await deepgram_stt.transcribe(audio_frames)\n logger.info(f\"STT result: {transcription}\")\n # e.g., \"Why is the pump pressure dropping below threshold?\"\n \n # ── Step 3: Understanding (NLU + RAG retrieval) ───────────────\n # Embed the query using Google Gemini\n query_vector = await gemini_embed(transcription)\n \n # Similarity search in MongoDB Atlas\n relevant_docs = await mongo_vector_search(query_vector, top_k=3)\n context = \"\\n\\n\".join([doc[\"text\"] for doc in relevant_docs])\n \n # ── Step 4: Reply Generation ──────────────────────────────────\n # Grok LLM receives query + retrieved context\n augmented_prompt = f\"Context:\\n{context}\\n\\nQuestion: {transcription}\"\n llm_response = await grok_llm.generate(augmented_prompt)\n logger.info(f\"LLM response: {llm_response}\")\n \n # ── Step 5: Text-to-Speech ────────────────────────────────────\n audio_response = await elevenlabs_tts.synthesize(llm_response)\n \n return audio_response # Streamed back via WaveSocket\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Quick Reference: Key Pipeline Values"}]},{"type":"table","content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Concept"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Typical Value"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Audio Frame Duration"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"≈ 0.1 seconds"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Embedding Model"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Google Gemini (768-dim vectors)"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Vector Store"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"MongoDB Atlas (vector index)"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"LLM Call Timeout"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"30 seconds"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Transport Protocol"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"WaveRTC / WebSocket"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Runner Loop Frequency"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"≈ 10Hz (10 cycles/second)"}]}]}]}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"6. Google Gemini Embeddings & MongoDB Atlas Vector Store"}]},{"type":"customImage","attrs":{"src":"https://67dc5f63b8b6a6f0ea39b99a.storage.fermion.app/public-cached/markdown-editor-file/file-unique-id/nijg0cfcftg8j8aefiy8bsuugpiacwux-q4pccs-6qogglugvaloadax7sfw7xfaeqgmybl8wartfrggc5kp0lefrsmdyyvb4aca","alt":null,"width":600,"height":327}},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"How Embeddings Power RAG"}]},{"type":"orderedList","attrs":{"start":1,"type":null},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Ingestion time:"},{"type":"text","text":" Documents are chunked → each chunk is embedded by Gemini into a 768-dim vector → stored in MongoDB Atlas"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Query time:"},{"type":"text","text":" User's transcribed question is embedded → MongoDB similarity search → top-K relevant chunks returned to LLM"}]}]}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Google Gemini Embedding Integration"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"type":"text","text":"import google.generativeai as genai\nimport os\n\n# Initialize Gemini client\ngenai.configure(api_key=os.getenv(\"GOOGLE_API_KEY\"))\n\nasync def gemini_embed(text: str) -> list[float]:\n \"\"\"\n Generate a 768-dimensional vector embedding for a piece of text.\n Used for both document ingestion and query-time similarity search.\n \"\"\"\n result = genai.embed_content(\n model=\"models/embedding-001\",\n content=text,\n task_type=\"retrieval_document\" # or \"retrieval_query\" for queries\n )\n return result[\"embedding\"] # list of 768 floats\n\n\nasync def embed_query(query: str) -> list[float]:\n \"\"\"Embed a user query for similarity search (different task type).\"\"\"\n result = genai.embed_content(\n model=\"models/embedding-001\",\n content=query,\n task_type=\"retrieval_query\"\n )\n return result[\"embedding\"]\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"MongoDB Atlas Vector Search"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"type":"text","text":"from pymongo import MongoClient\nimport os\n\nclient = MongoClient(os.getenv(\"MONGODB_URI\"))\ndb = client[os.getenv(\"DB_NAME\", \"rag_voice_db\")]\ncollection = db[\"equipment_chunks\"]\n\nasync def mongo_vector_search(query_vector: list[float], top_k: int = 3) -> list[dict]:\n \"\"\"\n Perform a cosine similarity search over the MongoDB Atlas vector index.\n Returns the top-K most relevant document chunks.\n \"\"\"\n pipeline = [\n {\n \"$vectorSearch\": {\n \"index\": \"vector_index\",\n \"path\": \"embedding\",\n \"queryVector\": query_vector,\n \"numCandidates\": 100,\n \"limit\": top_k\n }\n },\n {\n \"$project\": {\n \"text\": 1,\n \"chunk_id\": 1,\n \"document_id\": 1,\n \"equipment_id\": 1,\n \"score\": {\"$meta\": \"vectorSearchScore\"}\n }\n }\n ]\n \n results = list(collection.aggregate(pipeline))\n \n # Apply threshold — only return highly relevant results\n THRESHOLD = 0.75\n return [r for r in results if r.get(\"score\", 0) >= THRESHOLD]\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"MongoDB Atlas Vector Index Configuration"}]},{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Create this index in the Atlas UI under "},{"type":"text","marks":[{"type":"bold"}],"text":"Data Explorer → Indexes"},{"type":"text","text":":"}]},{"type":"codeBlock","attrs":{"language":"json"},"content":[{"type":"text","text":"{\n \"mappings\": {\n \"dynamic\": true,\n \"fields\": {\n \"embedding\": {\n \"dimensions\": 768,\n \"similarity\": \"cosine\",\n \"type\": \"knnVector\"\n }\n }\n }\n}\n"}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Region note:"},{"type":"text","text":" Select the AWS region closest to your deployment (e.g., Mumbai / "},{"type":"text","marks":[{"type":"code"}],"text":"ap-south-1"},{"type":"text","text":") to minimize latency between your ECS containers and MongoDB Atlas."}]}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"7. FastAPI: Equipment Routes vs Stream Endpoints"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Two Types of Endpoints"}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Equipment:"},{"type":"text","text":" Traditional FastAPI route handling (e.g., "},{"type":"text","marks":[{"type":"code"}],"text":"POST /equipment"},{"type":"text","text":"). Processes a "},{"type":"text","marks":[{"type":"bold"}],"text":"single request"},{"type":"text","text":" — creating, reading, or updating equipment records. Think of it as the backend code that handles one-shot operations."}]}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Stream:"},{"type":"text","text":" A continuous, bidirectional channel via WaveSocket that keeps the pipeline alive, allowing real-time audio chunks to flow in both directions simultaneously."}]}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Equipment Endpoint (Single Request)"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"type":"text","text":"# backend/main.py\nfrom fastapi import FastAPI\nfrom pydantic import BaseModel\nfrom typing import Optional\n\napp = FastAPI(title=\"RAG Voice Assistant API\", version=\"1.0.0\")\n\nclass EquipmentCreate(BaseModel):\n equipment_id: str\n name: str\n description: str\n documents: list[str] # Raw document texts to chunk and embed\n\nclass EquipmentResponse(BaseModel):\n equipment_id: str\n chunks_created: int\n status: str\n\n@app.post(\"/equipment\", response_model=EquipmentResponse)\nasync def create_equipment(payload: EquipmentCreate):\n \"\"\"\n Ingest documents for a piece of equipment.\n Chunks text, generates Gemini embeddings, and stores in MongoDB Atlas.\n This is the 'equipment' route — a single request that populates the vector store.\n \"\"\"\n total_chunks = 0\n \n for doc_text in payload.documents:\n chunks = chunk_document(doc_text)\n \n for idx, chunk in enumerate(chunks):\n embedding = await gemini_embed(chunk)\n \n await collection.insert_one({\n \"chunk_id\": f\"{payload.equipment_id}_{idx}\",\n \"document_id\": f\"doc_{idx}\",\n \"equipment_id\": payload.equipment_id,\n \"chunking_index\": idx,\n \"text\": chunk,\n \"embedding\": embedding,\n \"scoring\": None, # Populated at retrieval time\n \"threshold_value\": 0.75\n })\n total_chunks += 1\n \n return EquipmentResponse(\n equipment_id=payload.equipment_id,\n chunks_created=total_chunks,\n status=\"ingested\"\n )\n\n@app.get(\"/equipment/{equipment_id}\")\nasync def get_equipment(equipment_id: str):\n \"\"\"Retrieve equipment metadata and chunk count.\"\"\"\n count = await collection.count_documents({\"equipment_id\": equipment_id})\n return {\"equipment_id\": equipment_id, \"chunk_count\": count}\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Stream Endpoint (Real-Time Audio)"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"type":"text","text":"from fastapi import WebSocket, WebSocketDisconnect\nimport asyncio\n\n@app.websocket(\"/stream\")\nasync def voice_stream(websocket: WebSocket):\n \"\"\"\n Stream endpoint — continuous bidirectional audio channel.\n Keeps the voice pipeline alive for real-time conversation.\n This is wired to both MongoDB Atlas and PipeCat services.\n \"\"\"\n await websocket.accept()\n logger.info(\"WaveSocket connection established\")\n \n pipeline = MyRAGPipeline()\n \n try:\n while True:\n # Receive audio frame from frontend (≈0.1s of audio)\n audio_frame = await websocket.receive_bytes()\n \n # Process through the five-step pipeline\n audio_response = await pipeline.process_frame(audio_frame)\n \n # Stream synthesized audio back to frontend\n if audio_response:\n await websocket.send_bytes(audio_response)\n \n except WebSocketDisconnect:\n logger.info(\"Client disconnected from voice stream\")\n except Exception as e:\n logger.error(f\"Stream error: {e}\")\n await websocket.close()\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Both Endpoints Connected to the Same Services"}]},{"type":"codeBlock","attrs":{"language":"javascript"},"content":[{"type":"text","text":" ┌────────────────┐\nPOST /equipment ──▶ │ │ ──▶ MongoDB Atlas (insert chunks)\n │ FastAPI │\n │ main.py │ ──▶ PipeCat services (STT, LLM, TTS)\nWS /stream ──▶ │ │\n └────────────────┘\n"}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"8. Data Chunking & Indexing Schema"}]},{"type":"customImage","attrs":{"src":"https://67dc5f63b8b6a6f0ea39b99a.storage.fermion.app/public-cached/markdown-editor-file/file-unique-id/4zfhrq3_bahlyco68k890pgmovybjtctsnqidg7u4gwi5o6ihshauug4h9wzvs6cmpty_zqx17w63bkyokr0tox1bu-e6co1tllu","alt":null,"width":600,"height":327}},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Why Chunking Matters"}]},{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"An entire document cannot be embedded as one unit — the embedding model has a token limit, and a single vector for a 50-page manual would lose all fine-grained detail. Chunking splits the document into overlapping pieces, each embedded separately."}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Chunking Logic"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"type":"text","text":"def chunk_document(\n text: str,\n chunk_size: int = 500, # characters per chunk\n overlap: int = 50 # overlap between consecutive chunks\n) -> list[str]:\n \"\"\"\n Split a document into overlapping chunks for embedding.\n Overlap preserves context at chunk boundaries.\n \"\"\"\n chunks = []\n start = 0\n \n while start < len(text):\n end = start + chunk_size\n chunk = text[start:end]\n \n # Avoid splitting mid-sentence — walk back to last period\n if end < len(text) and '.' in chunk:\n last_period = chunk.rfind('.')\n if last_period > chunk_size // 2: # Only trim if boundary is reasonable\n chunk = chunk[:last_period + 1]\n end = start + last_period + 1\n \n chunks.append(chunk.strip())\n start = end - overlap # Slide window with overlap\n \n return [c for c in chunks if len(c) > 50] # Filter tiny fragments\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"MongoDB Document Schema for Chunks"}]},{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Each chunk stored in MongoDB Atlas has these fields (populated by the "},{"type":"text","marks":[{"type":"code"}],"text":"create_equipment"},{"type":"text","text":" function in "},{"type":"text","marks":[{"type":"code"}],"text":"bot.py"},{"type":"text","text":"):"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"type":"text","text":"# Schema for each stored chunk document\nchunk_document = {\n # Identity fields\n \"chunk_id\": \"pump_manual_42\", # Unique ID for this text segment\n \"document_id\": \"pump_manual_v2\", # Links chunk back to source document\n \"equipment_id\": \"centrifugal_pump_01\", # Originating hardware/dataset\n \"chunking_index\": 42, # Order of chunk within its document\n \n # Content\n \"text\": \"The pump should be inspected...\", # Raw chunk text\n \n # Vector\n \"embedding\": [0.234, -0.812, ..., 0.091], # 768-dim Gemini vector\n \n # Retrieval metadata\n \"scoring\": 0.89, # Cosine similarity score (populated at query time)\n \"threshold_value\": 0.75, # Minimum score to consider this chunk relevant\n}\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Threshold Filtering"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"type":"text","text":"# Only surface chunks above the relevance threshold\nTHRESHOLD = 0.75\n\nresults = await mongo_vector_search(query_vector, top_k=10)\nrelevant = [r for r in results if r[\"score\"] >= THRESHOLD]\n\n# If nothing meets the threshold — graceful fallback\nif not relevant:\n return \"I don't have specific information about that in my knowledge base.\"\n"}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"9. Project Structure & Core Code Walkthrough"}]},{"type":"codeBlock","attrs":{"language":"javascript"},"content":[{"type":"text","text":"pipcat-quickstart/\n├── backend/\n│ ├── main.py # FastAPI entry point — equipment & stream routes\n│ ├── bot.py # RAG pipeline — registers STT, LLM, TTS tasks\n│ └── logger.py # Centralized logging configuration\n├── frontend/\n│ ├── src/\n│ │ ├── App.jsx # Root React component\n│ │ └── VoiceChat.jsx # WaveSocket audio capture & playback\n│ ├── package.json\n│ └── tailwind.config.js\n├── env/\n│ └── .env # All API keys and DB credentials (git-ignored)\n├── Dockerfile # Container image definition\n├── docker-compose.yml # Multi-service orchestration\n└── .github/\n └── workflows/\n └── deploy.yml # CI/CD: build → ECR → ECS/Fargate\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","marks":[{"type":"code"}],"text":"bot.py"},{"type":"text","text":" Key Responsibilities"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"type":"text","text":"# backend/bot.py — RAG pipeline registration\n\nimport os\nimport logging\nfrom dotenv import load_dotenv\n\n# PyDentic — custom helper class implementing project-specific logic\n# (equipment creation, chunking schema, metadata extraction)\nfrom pydentic import EquipmentHelper\n\nload_dotenv()\nlogger = logging.getLogger(__name__)\n\n# ── Service initialization ────────────────────────────────────────\nstt_service = DeepgramSTTService(api_key=os.getenv(\"DEEPGRAM_API_KEY\"))\nllm_service = GrokLLMService(api_key=os.getenv(\"GROK_API_KEY\"))\ntts_service = ElevenLabsTTSService(api_key=os.getenv(\"ELEVENLABS_API_KEY\"))\n\n# ── PyDentic helper for equipment ingestion ───────────────────────\nequipment_helper = EquipmentHelper(\n mongo_uri=os.getenv(\"MONGODB_URI\"),\n db_name=os.getenv(\"DB_NAME\"),\n embed_fn=gemini_embed # Google Gemini embedding function\n)\n\n# ── Register pipeline tasks ───────────────────────────────────────\npipeline = Pipeline([\n transport.input(),\n stt_service,\n rag_processor, # Vector retrieval via MongoDB Atlas\n llm_context_aggregator.user(),\n llm_service,\n tts_service,\n transport.output(),\n llm_context_aggregator.assistant(),\n])\n"}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"PyDentic"},{"type":"text","text":" is the project-specific custom Python code that implements equipment object creation and handles the chunking schema. When adapting the base example from OpenAI to Grok, replace the OpenAI client import with the Grok SDK and update the API key reference from "},{"type":"text","marks":[{"type":"code"}],"text":"OPENAI_API_KEY"},{"type":"text","text":" to "},{"type":"text","marks":[{"type":"code"}],"text":"GROK_API_KEY"},{"type":"text","text":"."}]}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","marks":[{"type":"code"}],"text":"logger.py"},{"type":"text","text":" Centralized Logging"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"type":"text","text":"# backend/logger.py\nimport logging\nimport sys\n\ndef get_logger(name: str) -> logging.Logger:\n \"\"\"Returns a configured logger instance for any module.\"\"\"\n logger = logging.getLogger(name)\n \n if not logger.handlers:\n handler = logging.StreamHandler(sys.stdout)\n handler.setFormatter(logging.Formatter(\n \"%(asctime)s | %(levelname)-8s | %(name)s | %(message)s\"\n ))\n logger.addHandler(handler)\n logger.setLevel(logging.INFO)\n \n return logger\n\n# Usage in any module:\n# from logger import get_logger\n# logger = get_logger(__name__)\n"}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"10. Environment Setup & API Key Configuration"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Required Services & Keys"}]},{"type":"table","content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Service"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Purpose"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"code"}],"text":".env"},{"type":"text","text":" Variable"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"MongoDB Atlas"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Vector database for RAG retrieval"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"code"}],"text":"MONGODB_URI"},{"type":"text","text":", "},{"type":"text","marks":[{"type":"code"}],"text":"DB_NAME"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Deepgram"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Speech-to-Text (STT)"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"code"}],"text":"DEEPGRAM_API_KEY"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"ElevenLabs (11Labs)"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Text-to-Speech (TTS)"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"code"}],"text":"ELEVENLABS_API_KEY"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Google Gemini"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Embedding generation"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"code"}],"text":"GOOGLE_API_KEY"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Grok"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"LLM generation"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"code"}],"text":"GROK_API_KEY"}]}]}]}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"How to Obtain Each Key"}]},{"type":"table","content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Service"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Steps"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"ElevenLabs"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Sign up → Developer → API Key → name it "},{"type":"text","marks":[{"type":"code"}],"text":"rag_ai_agent"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Deepgram"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Sign up → Create Project → name containing "},{"type":"text","marks":[{"type":"code"}],"text":"rag"},{"type":"text","text":" → generate API key"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Grok"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Use existing Grok key (already assigned in the repo)"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Google Gemini"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Create API key under Embedding → label "},{"type":"text","marks":[{"type":"code"}],"text":"gemini_embedding"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"MongoDB Atlas"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null},"content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"Sign up → create free cluster → note username & password → configure connection string"}]}]}]}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","marks":[{"type":"code"}],"text":".env"},{"type":"text","text":" File"}]},{"type":"codeBlock","attrs":{"language":"bash"},"content":[{"type":"text","text":"# env/.env — NEVER commit this file to git\n# Add env/ to .gitignore\n\nMONGODB_URI=mongodb+srv://username:password@cluster.mongodb.net/?retryWrites=true\nDB_NAME=rag_voice_db\n\nDEEPGRAM_API_KEY=dg_abc123yourkeyhere\nELEVENLABS_API_KEY=sk_abc123yourkeyhere\nGOOGLE_API_KEY=AIzaSyAbc123yourkeyhere\nGROK_API_KEY=gsk_abc123yourkeyhere\n"}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Security rule:"},{"type":"text","text":" Store every key in "},{"type":"text","marks":[{"type":"code"}],"text":".env"},{"type":"text","text":". Add "},{"type":"text","marks":[{"type":"code"}],"text":"env/"},{"type":"text","text":" and "},{"type":"text","marks":[{"type":"code"}],"text":".env"},{"type":"text","text":" to "},{"type":"text","marks":[{"type":"code"}],"text":".gitignore"},{"type":"text","text":". In production, these values are injected by "},{"type":"text","marks":[{"type":"bold"}],"text":"AWS Secrets Manager"},{"type":"text","text":" — never hardcode credentials anywhere in source code."}]}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"11. MongoDB Atlas Vector Store Setup"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Step-by-Step Atlas Configuration"}]},{"type":"codeBlock","attrs":{"language":"javascript"},"content":[{"type":"text","text":"1. Sign up at cloud.mongodb.com\n │\n ▼\n2. Create a free cluster\n → Select AWS region: ap-south-1 (Mumbai) or ap-southeast-1 (Singapore)\n → Free tier: 512MB storage — sufficient for development\n │\n ▼\n3. Create a database user\n → Security → Database Access → Add New Database User\n → Username + strong password\n → Role: readWrite on your database\n │\n ▼\n4. Whitelist your IP\n → Security → Network Access → Add IP Address\n → For development: Allow Access from Anywhere (0.0.0.0/0)\n → For production: restrict to ECS task IP ranges\n │\n ▼\n5. Get the connection string\n → Cluster → Connect → Connect your application\n → Copy: mongodb+srv://user:pass@cluster.xyz.mongodb.net/\n │\n ▼\n6. Create the vector index\n → Data Explorer → Select collection → Indexes tab → Create Search Index\n → Use the JSON configuration below\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Vector Index JSON"}]},{"type":"codeBlock","attrs":{"language":"json"},"content":[{"type":"text","text":"{\n \"mappings\": {\n \"dynamic\": true,\n \"fields\": {\n \"embedding\": {\n \"dimensions\": 768,\n \"similarity\": \"cosine\",\n \"type\": \"knnVector\"\n }\n }\n }\n}\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Test the Connection"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"type":"text","text":"from pymongo import MongoClient\nimport os\nfrom dotenv import load_dotenv\n\nload_dotenv()\n\ndef test_atlas_connection():\n client = MongoClient(os.getenv(\"MONGODB_URI\"))\n try:\n client.admin.command(\"ping\")\n print(\"✅ MongoDB Atlas connection successful\")\n db = client[os.getenv(\"DB_NAME\")]\n print(f\"✅ Database: {db.name}\")\n print(f\"✅ Collections: {db.list_collection_names()}\")\n except Exception as e:\n print(f\"❌ Connection failed: {e}\")\n finally:\n client.close()\n\ntest_atlas_connection()\n"}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"12. Full Implementation: From Clone to Running Locally"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Complete Setup Sequence"}]},{"type":"codeBlock","attrs":{"language":"bash"},"content":[{"type":"text","text":"# ── Step 1: Clone repository ──────────────────────────────────────\ngit clone https://github.com/your-org/pipcat-quickstart.git\ncd pipcat-quickstart\n\n# ── Step 2: Install uv (mandatory — pipeline cannot build without it) ──\npip install uv\n\n# ── Step 3: Initialize PipeCat orchestration ─────────────────────\npipcat qk start\n\n# ── Step 4: Set up environment variables ──────────────────────────\ncp environment.example env/.env\n# Edit env/.env — fill in all five API keys and MONGODB_URI\n\n# ── Step 5: Install all dependencies ──────────────────────────────\nuv sync\n\n# ── Step 6: Start the backend ─────────────────────────────────────\ncd backend\nuvicorn main:app --reload\n# ✅ Expected: \"Uvicorn running on http://0.0.0.0:8000\"\n\n# ── Step 7: Start the frontend (new terminal) ─────────────────────\ncd frontend\nnpm install\nnpm start\n# ✅ Expected: \"Compiled successfully! Now listening on port 3000\"\n\n# ── Step 8: Build with Docker Compose ─────────────────────────────\ndocker compose up --build\n# Builds both backend and frontend containers and starts them together\n\n# ── Step 9: Verify end-to-end flow ────────────────────────────────\n# Open http://localhost:3000\n# Click Connect, speak into microphone\n# Confirm: STT transcription appears → LLM responds → TTS plays audio\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Docker Compose Configuration"}]},{"type":"codeBlock","attrs":{"language":"yaml"},"content":[{"type":"text","text":"# docker-compose.yml\nversion: \"3.9\"\n\nservices:\n backend:\n build:\n context: ./backend\n dockerfile: Dockerfile\n ports:\n - \"8000:8000\"\n - \"8765:8765\" # WaveSocket port\n env_file:\n - ./env/.env\n volumes:\n - ./backend:/app\n command: uvicorn main:app --host 0.0.0.0 --port 8000 --reload\n\n frontend:\n build:\n context: ./frontend\n dockerfile: Dockerfile\n ports:\n - \"3000:3000\"\n environment:\n - REACT_APP_API_URL=http://localhost:8000\n - REACT_APP_WS_URL=ws://localhost:8765\n depends_on:\n - backend\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"Frontend: React WaveSocket Integration"}]},{"type":"codeBlock","attrs":{"language":"jsx"},"content":[{"type":"text","text":"// frontend/src/VoiceChat.jsx\nimport { useState, useRef, useEffect } from \"react\";\n\nconst WS_URL = process.env.REACT_APP_WS_URL || \"ws://localhost:8765/stream\";\n\nexport default function VoiceChat() {\n const [connected, setConnected] = useState(false);\n const [transcript, setTranscript] = useState([]);\n const wsRef = useRef(null);\n const mediaRecorderRef = useRef(null);\n\n const connect = async () => {\n // Open WaveSocket connection to the /stream endpoint\n wsRef.current = new WebSocket(WS_URL);\n \n wsRef.current.onopen = () => {\n setConnected(true);\n startAudioCapture();\n };\n \n wsRef.current.onmessage = async (event) => {\n // Receive synthesized audio from ElevenLabs via backend\n const audioBlob = new Blob([event.data], { type: \"audio/wav\" });\n const audioUrl = URL.createObjectURL(audioBlob);\n const audio = new Audio(audioUrl);\n await audio.play();\n };\n \n wsRef.current.onclose = () => setConnected(false);\n };\n\n const startAudioCapture = async () => {\n const stream = await navigator.mediaDevices.getUserMedia({ audio: true });\n mediaRecorderRef.current = new MediaRecorder(stream);\n \n mediaRecorderRef.current.ondataavailable = (event) => {\n // Send ≈0.1s audio frames to the backend stream endpoint\n if (event.data.size > 0 && wsRef.current?.readyState === WebSocket.OPEN) {\n wsRef.current.send(event.data);\n }\n };\n \n // Emit data every 100ms = 0.1s frames\n mediaRecorderRef.current.start(100);\n };\n\n const disconnect = () => {\n mediaRecorderRef.current?.stop();\n wsRef.current?.close();\n setConnected(false);\n };\n\n return (\n

RAG Voice Assistant

\n \n \n {connected ? \"Disconnect\" : \"Connect & Speak\"}\n \n \n

\n {transcript.map((msg, i) => (\n

\n {msg.role === \"user\" ? \"You\" : \"Assistant\"}: {msg.text}\n

\n ))}\n

\n );\n}\n"}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"13. CI/CD Pipeline: GitHub Actions → ECR → ECS/Fargate"}]},{"type":"customImage","attrs":{"src":"https://67dc5f63b8b6a6f0ea39b99a.storage.fermion.app/public-cached/markdown-editor-file/file-unique-id/730tymvofb-xqaszzdnvpv7k0awwuerjn_gddl6foufnbkiyvb_6hl6uoc3kjwouhhk9hlg9lo_tsocziw8bsotn2-nug3ujyf2g","alt":null,"width":600,"height":327}},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"High-Level Deployment Architecture"}]},{"type":"codeBlock","attrs":{"language":"javascript"},"content":[{"type":"text","text":"[Developer pushes to main branch]\n │\n ▼\n[GitHub Actions triggered]\n │\n ┌─────┴────────────────────────────────────┐\n │ 1. Checkout code │\n │ 2. Set up Python 3.11 │\n │ 3. uv sync → install deps │\n │ 4. Run test suite │\n │ 5. Build Docker image │\n │ 6. Push image to AWS ECR │\n │ 7. Update ECS task definition │\n │ 8. Deploy to ECS/Fargate │\n └──────────────────────────────────────────┘\n │\n ▼\n[AWS ECS/Fargate — containerized backend]\n │\n[Application Load Balancer (ALB)]\n │ routes traffic\n ▼\n[Multiple ECS task instances]\n │\n[CloudWatch — logs & metrics]\n[Secrets Manager — API keys at runtime]\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"GitHub Actions Workflow"}]},{"type":"codeBlock","attrs":{"language":"yaml"},"content":[{"type":"text","text":"# .github/workflows/deploy.yml\nname: Deploy RAG Voice Assistant\n\non:\n push:\n branches: [main]\n\nenv:\n AWS_REGION: ap-south-1 # Mumbai — matches MongoDB Atlas region\n ECR_REPOSITORY: rag-voice-agent\n ECS_SERVICE: rag-voice-service\n ECS_CLUSTER: rag-voice-cluster\n\njobs:\n test-and-deploy:\n runs-on: ubuntu-latest\n \n steps:\n - name: Checkout code\n uses: actions/checkout@v3\n\n - name: Set up Python 3.11\n uses: actions/setup-python@v4\n with:\n python-version: \"3.11\"\n\n - name: Install dependencies\n run: |\n pip install uv\n cd backend && uv sync\n\n - name: Run tests\n run: |\n cd backend && uv run pytest tests/ -v\n env:\n MONGODB_URI: ${{ secrets.MONGODB_URI }}\n DEEPGRAM_API_KEY: ${{ secrets.DEEPGRAM_API_KEY }}\n\n - name: Configure AWS credentials\n uses: aws-actions/configure-aws-credentials@v2\n with:\n aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}\n aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n aws-region: ${{ env.AWS_REGION }}\n\n - name: Login to Amazon ECR\n id: login-ecr\n uses: aws-actions/amazon-ecr-login@v1\n\n - name: Build, tag, and push Docker image to ECR\n env:\n ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}\n IMAGE_TAG: ${{ github.sha }}\n run: |\n # Build backend image\n docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG ./backend\n docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG\n \n # Tag as latest for convenience\n docker tag $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG \\\n $ECR_REGISTRY/$ECR_REPOSITORY:latest\n docker push $ECR_REGISTRY/$ECR_REPOSITORY:latest\n\n - name: Deploy to Amazon ECS\n uses: aws-actions/amazon-ecs-deploy-task-definition@v1\n with:\n task-definition: task-definition.json\n service: ${{ env.ECS_SERVICE }}\n cluster: ${{ env.ECS_CLUSTER }}\n wait-for-service-stability: true\n"}]},{"type":"heading","attrs":{"textAlign":null,"level":3},"content":[{"type":"text","text":"ECS Task Definition with Secrets Manager"}]},{"type":"codeBlock","attrs":{"language":"json"},"content":[{"type":"text","text":"{\n \"family\": \"rag-voice-agent\",\n \"networkMode\": \"awsvpc\",\n \"requiresCompatibilities\": [\"FARGATE\"],\n \"cpu\": \"512\",\n \"memory\": \"1024\",\n \"containerDefinitions\": [\n {\n \"name\": \"rag-backend\",\n \"image\": \".dkr.ecr.ap-south-1.amazonaws.com/rag-voice-agent:latest\",\n \"portMappings\": [\n {\"containerPort\": 8000, \"protocol\": \"tcp\"},\n {\"containerPort\": 8765, \"protocol\": \"tcp\"}\n ],\n \"secrets\": [\n {\"name\": \"MONGODB_URI\", \"valueFrom\": \"arn:aws:secretsmanager:...\"},\n {\"name\": \"DEEPGRAM_API_KEY\", \"valueFrom\": \"arn:aws:secretsmanager:...\"},\n {\"name\": \"ELEVENLABS_API_KEY\", \"valueFrom\": \"arn:aws:secretsmanager:...\"},\n {\"name\": \"GOOGLE_API_KEY\", \"valueFrom\": \"arn:aws:secretsmanager:...\"},\n {\"name\": \"GROK_API_KEY\", \"valueFrom\": \"arn:aws:secretsmanager:...\"}\n ],\n \"logConfiguration\": {\n \"logDriver\": \"awslogs\",\n \"options\": {\n \"awslogs-group\": \"/ecs/rag-voice-agent\",\n \"awslogs-region\": \"ap-south-1\",\n \"awslogs-stream-prefix\": \"ecs\"\n }\n }\n }\n ]\n}\n"}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"14. Real-Time Call Center Use Case"}]},{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"The same architecture applies directly to a "},{"type":"text","marks":[{"type":"bold"}],"text":"real-time industrial support call center"},{"type":"text","text":":"}]},{"type":"codeBlock","attrs":{"language":"javascript"},"content":[{"type":"text","text":"[Machine Operator] ──speaks──▶ [WaveRTC client on operator's headset]\n │\n [Backend pipeline]\n │\n ┌───────────────┴────────────────┐\n │ │\n [Deepgram STT] [MongoDB Atlas]\n \"pump pressure Equipment manuals\n dropping below Maintenance guides\n threshold\" Troubleshooting docs\n │ │\n └──────────────┬─────────────────┘\n │ enriched context\n ▼\n [Grok LLM]\n \"Check the inlet valve —\n likely cavitation.\n Reduce flow rate by 20%.\"\n │\n [ElevenLabs TTS]\n │\n [Synthesized audio → operator's headset]\n"}]},{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"The pipeline operates in a "},{"type":"text","marks":[{"type":"bold"}],"text":"downstream"},{"type":"text","text":" (AI → human) and "},{"type":"text","marks":[{"type":"bold"}],"text":"upstream"},{"type":"text","text":" (human → AI) loop — exactly mirroring a live support call, but with the knowledge base of every equipment manual the company has ever produced."}]},{"type":"horizontalRule"},{"type":"heading","attrs":{"textAlign":null,"level":2},"content":[{"type":"text","text":"Key Takeaways"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"The "},{"type":"text","marks":[{"type":"bold"}],"text":"five-step pipeline"},{"type":"text","text":" (Listen → STT → Understand → Generate → TTS) is the core processing model; every component maps to one step."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Google Gemini embeddings"},{"type":"text","text":" (768-dim) combined with "},{"type":"text","marks":[{"type":"bold"}],"text":"MongoDB Atlas vector search"},{"type":"text","text":" power the RAG retrieval — the mechanism that keeps answers grounded in real documents."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"FastAPI has two endpoint types:"},{"type":"text","text":" "},{"type":"text","marks":[{"type":"code"}],"text":"equipment"},{"type":"text","text":" routes for one-shot data ingestion and the "},{"type":"text","marks":[{"type":"code"}],"text":"stream"},{"type":"text","text":" endpoint (WaveSocket) for real-time bidirectional audio."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Data chunking"},{"type":"text","text":" with proper schema (chunk_id, document_id, equipment_id, chunking_index, threshold_value) is what makes retrieval both fast and precise."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"The "},{"type":"text","marks":[{"type":"bold"}],"text":"Transport → Processor → Observer"},{"type":"text","text":" architecture cleanly separates networking, processing, and monitoring concerns — making the system easier to debug and extend."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Audio frames of ≈0.1s"},{"type":"text","text":" are the key to low latency — never send full recordings; always stream small chunks."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"AWS Secrets Manager"},{"type":"text","text":" is the production equivalent of "},{"type":"text","marks":[{"type":"code"}],"text":".env"},{"type":"text","text":" — all five API keys are injected at runtime into ECS task containers, never baked into the image."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","text":"The full "},{"type":"text","marks":[{"type":"bold"}],"text":"CI/CD chain"},{"type":"text","text":" (GitHub Actions → ECR → ECS/Fargate + ALB) ensures every push to "},{"type":"text","marks":[{"type":"code"}],"text":"main"},{"type":"text","text":" automatically builds, tests, and deploys without manual intervention."}]}]}]},{"type":"horizontalRule"},{"type":"paragraph","attrs":{"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"This blog is the fourth in the series on building a production RAG voice AI agent. It covers the complete system design — from RTVI architecture and Gemini embeddings through FastAPI equipment/stream endpoints, chunking schema, and full AWS ECS/Fargate deployment with CloudWatch monitoring."}]}]}