🎙️ SentimentRAG

AI-Powered CRM Conversation Intelligence Platform

Transcribe → Diarize → Analyze Sentiment → Fact-Check with RAG → Chat — an end-to-end pipeline for turning raw university counselling call recordings into structured, actionable intelligence.

Python 3.10+ | Flask 3.x | WhisperX GPU | ChromaDB | Firebase Auth | License Unspecified

---

🧠 What is this?

SentimentRAG listens to recorded university admission counselling calls, turns them into a speaker-labelled transcript, analyzes the sentiment of the conversation, and answers student questions about courses, fees, hostel, and scholarships using a Retrieval-Augmented Generation (RAG) chatbot grounded in real university brochure data — instead of letting an LLM hallucinate answers.

Built as part of an internship project for Better Study, a Career Point University initiative.

On top of transcription, the platform surfaces:

• Full speaker-diarized transcripts (SPEAKER_00, SPEAKER_01, ...)
• Sentiment breakdown (Positive / Neutral / Negative)
• Lead scoring and recommended follow-up actions
• A RAG-powered "Ask AI" assistant answering student queries from real hostel/scholarship/course data

---

✨ Features

Capability	Description
🗣️ Speech-to-Text + Diarization	Uses WhisperX (medium model) to transcribe, word-align, and label speakers — with anti-hallucination tuning (condition_on_previous_text=False, tuned silence/compression thresholds) for clean Hindi/Hinglish output.
🧩 Word-Level Alignment	Wav2Vec2-based forced alignment for accurate per-word timestamps, required for reliable diarization.
🎭 Speaker Diarization	Pyannote-based diarization pipeline identifies who spoke when in each call.
💬 Sentiment Analysis	Multilingual sentiment classification (Hindi-capable) over the transcript, aggregated into an overall call sentiment score.
📚 RAG Fact-Checking Chatbot	"Ask AI" answers student questions (fees, hostel, scholarships, courses) by retrieving relevant chunks from a ChromaDB vector store built from official university brochure data — not from the LLM's memory.
🎯 Lead Scoring	Generates a 0–100 lead-quality score with supporting rationale for each call.
📊 Analytics Dashboard	Sentiment trends, lead score distribution, and SOP compliance visualized with Chart.js.
🔐 Firebase Authentication	Email/password signup & login backed by Firebase Admin SDK + Firestore.
🖥️ Full Frontend	Multi-page dashboard (Dashboard, Upload, Ask AI, Analytics, Report) — plain HTML/CSS/JS, served directly by Flask.

---

🏗️ Architecture & Pipeline

🎙️ Audio Call (Upload)
        │
        ▼
┌─────────────────────┐
│  WhisperX (GPU)      │  → Transcription + Word Alignment + Diarization
└─────────────────────┘
        │
        ▼
┌─────────────────────┐
│  Sentiment Analysis   │  → Positive / Neutral / Negative + confidence
└─────────────────────┘
        │
        ▼
┌─────────────────────┐
│  RAG Retriever        │  → ChromaDB + Sentence-Transformer embeddings
│  (Chroma + LLM)       │     fetch matching university brochure facts
└─────────────────────┘
        │
        ▼
📄 Report (Transcript + Sentiment + Lead Score + Recommendations)
        +
💬 Ask AI Chat (grounded Q&A over university data)

---

🛠️ Tech Stack

Category	Technology
Backend	Python, Flask, Flask-CORS
Authentication	Firebase Admin SDK (Auth + Firestore)
Speech-to-Text	WhisperX (medium, GPU, int8_float16)
Alignment	Wav2Vec2 (forced alignment)
Diarization	Pyannote Audio (speaker-diarization-community-1)
Sentiment Analysis	cardiffnlp/twitter-xlm-roberta-base-sentiment (multilingual, Hindi-capable)
RAG / Vector Store	ChromaDB + Sentence-Transformers embeddings
LLM	Google Gemini
Frontend	HTML5, CSS3, Vanilla JS, Chart.js
Data Storage	JSON transcript store, ChromaDB persistent store

---

📁 Project Structure

SentimentRAG/
├── app.py                      # Flask app — routes, upload handling, auth, RAG endpoint
├── config.py                   # App configuration
├── transcription.py            # WhisperX transcription + alignment + diarization pipeline
├── sentiment_analysis.py       # Multilingual sentiment analysis over transcripts
├── rag_retriever.py            # RAG query pipeline (ChromaDB retrieval + LLM answer generation)
├── vector_store.py             # ChromaDB client setup
├── embedding_hostel.py          # Ingests hostel fee/data into ChromaDB
├── embedding_scholarship.py     # Ingests scholarship data into ChromaDB
├── test_rag.py                  # RAG pipeline test script
├── requirements.txt             # Python dependencies
├── .env.example                 # Environment variable template
├── frontend/
│   ├── index.html               # Landing page
│   ├── login.html / signup.html # Auth pages
│   ├── dashboard.html           # Overview stats + recent calls
│   ├── upload.html              # Drag & drop call recording upload
│   ├── report.html              # Per-call transcript + sentiment + lead score report
│   ├── chat.html                # "Ask AI" RAG chatbot interface
│   ├── analytics.html           # Sentiment/lead-score/SOP trend charts
│   └── assets/                  # Shared CSS & JS
└── README.md

---

🔌 API Reference

Method	Endpoint	Description
GET	/	Serves the frontend landing page
GET	/api/health	Basic health check
POST	/api/upload	Upload an audio file (multipart/form-data: audio). Runs transcription + diarization and stores the result.
GET	/api/transcripts	Lists all processed call transcripts
GET	/api/transcript/<id>	Fetch a single transcript/report by ID
POST	/api/ask	Ask a question ({"question": "...", "category": "..."}) — answered via RAG over university brochure data
POST	/api/auth/signup	Create a new user account (Firebase)
POST	/api/auth/login	Log in an existing user

---

🚀 Getting Started

Prerequisites

• Python 3.10+
• NVIDIA GPU with CUDA support (recommended — transcription defaults to GPU; can be reconfigured for CPU with a smaller Whisper model)
• A Firebase project with a serviceAccountKey.json
• A Google Gemini API key (with billing enabled to avoid free-tier quota limits)
• A Hugging Face account + token (accept the pyannote diarization model terms before running)

1. Clone & set up a virtual environment

git clone https://github.com/prabhattripathi9351/rag_based_internship_project.git
cd rag_based_internship_project

python -m venv .venv

# Windows
.venv\Scripts\Activate.ps1

# macOS / Linux
source .venv/bin/activate

pip install -r requirements.txt

2. Configure environment variables

Copy .env.example to .env and fill in:

GEMINI_API_KEY=your_gemini_api_key
HF_TOKEN=your_huggingface_token

Also place your Firebase serviceAccountKey.json in the project root.

💡 Accept the relevant model terms on Hugging Face for the pyannote diarization models before running — otherwise diarization will fail with an auth error.

3. Build the knowledge base

Run the embedding scripts once to populate ChromaDB with university data:

python embedding_hostel.py
python embedding_scholarship.py

4. Run the app

python app.py

The app will be live at http://127.0.0.1:5000

---

🗺️ Roadmap Ideas

• Configurable CPU fallback for transcription (deploy without requiring a GPU)
• Persistent lead scoring & sentiment breakdown computed dynamically per call (currently partially static)
• Call duration & audio metadata extraction
• Admin dashboard for reviewing flagged/low-confidence transcripts
• Streaming transcription progress over WebSockets
• Dockerized deployment

---

🤝 Contributing

Issues and pull requests are welcome!

---

Built as part of an internship project for Better Study — a Career Point University initiative.