Commit 571efe

2025-10-14 08:38:51 Prashant Kumar: Update HLD
Projects/SkillShift/HLD.md ..
@@ 1,1 1,238 @@
# HLD
+
+
+ # SkillShift — System Overview
+
+ ## 1. Introduction
+
+ **SkillShift** is a system designed to help employees and HR/Admins manage, explore, and retrieve organizational knowledge through **AI-assisted interactions**.
+
+ Organizations often face operational and productivity risks when employees leave or move across teams without structured handovers or visibility into their unique knowledge, decisions, and expertise.
+
+ This tool leverages **NLP-based knowledge mining** to automatically build a dynamic knowledge base and organizational decisions — all derived from communication channels like **Google Chat** and **Telegram**. The platform centralizes internal communication data, maintains tags for context classification, and provides employees and HR/Admins with an AI assistant to query messages and insights related to them or their assigned tags.
+
+ In modern organizations, large volumes of valuable information — discussions, decisions, technical clarifications, and project context — are scattered across internal messaging tools. This tool provides a unified way to collect, organize, and retrieve such communication efficiently using AI-assisted search and tag-based categorization.
+
+ Unlike traditional HR or knowledge base systems, this assistant focuses on **conversation intelligence** rather than static documentation. Instead of manually searching through thousands of chat messages, users can simply ask **natural language questions** through the AI assistant, which semantically understands the context and retrieves relevant messages tagged to their area, project, or topic.
+
+ ### Platform Users
+
+ - **Employees**
+ - View their profiles, assigned tags, and interact with the AI assistant to find relevant discussions or information.
+ - **HR/Admins**
+ - Manage and assign tags, configure data source integrations, and use the AI assistant for organization-wide insights.
+
+ ---
+
+ ## 2. Objectives
+
+ The system aims to achieve the following core objectives:
+
+ - Consolidate organizational conversations from multiple data sources such as **Google Chat** and **Telegram** into a single, searchable knowledge base.
+ - Simplify knowledge discovery through a **conversational AI assistant**.
+ - Enable structured organization of communication via **tags** for projects, discussions, or themes.
+ - Empower employees with **role-based access** to only relevant messages and topics.
+ - Provide HR/Admins tools for efficient management of users, tags, and data sources.
+ - Enhance organizational transparency and reduce information silos through AI-assisted knowledge retrieval.
+
+ ---
+
+ ## 3. Initial Scope (MVP Deliverables)
+
+ - **Integrations:** Google Chat & Telegram
+ - **Screens:**
+ - Employee Profile
+ - AI Assistant
+ - Tag Management
+ - Data Source Management
+
+ ---
+
+ ## 4. Users & Roles
+
+ - **Employee:** View their profile and access the AI assistant.
+ - **HR/Manager/System Admin:** Manage integrations and tag management.
+
+ ---
+
+ ## 5. Architecture Overview
+
+ *(Architecture diagram to be added in the final documentation.)*
+
+ ---
+
+ ## 6. Module Breakdown
+
+ ### 6.1 Integration Layer
+ - **Telegram Bot API:** Captures chat messages.
+ - **Google Chat API:** Webhook/polling for spaces & groups.
+
+ ---
+
+ ### 6.2 Processing & Embedding Layer
+ - **Text Preprocessing:** Removes stopwords, links, emojis, and Personally Identifiable Information (PII).
+ - **Embeddings:** Creates message text embeddings for semantic search.
+
+ ---
+
+ ### 6.3 Knowledge Store (PostgreSQL + pgvector)
+
+ #### Schema (MVP)
+ | Table | Description |
+ |-------|--------------|
+ | `employees` | id, external_id, name, role, dept, tenure, is_active, is_admin |
+ | `messages` | id, employee_id, data_source_id, text, embedding |
+ | `tags` | id, tag_name, is_active |
+ | `employee_tag` | id, tag_id, employee_id, is_active |
+ | `data_sources` | id, platform, external_id, name, tag_id, description, is_active |
+
+ #### Logical Data Model
+
+ - **Employees**
+ - Represents all individuals using or referenced by the platform.
+ - Attributes: `id`, `name`, `role`, `department`, `tenure`, `is_active`.
+ - Linked to multiple tags and messages.
+
+ - **Messages**
+ - Stores chat data from Google Chat or Telegram.
+ - Each message is tied to one employee and one data source.
+ - Stores cleaned text and a semantic embedding vector.
+
+ - **Tags**
+ - Represents thematic or organizational labels (e.g., *Project Alpha*, *DevOps*).
+ - Used to group employees and messages under meaningful categories.
+
+ - **Data Sources**
+ - Represents external chat systems (Telegram group, Google Chat space).
+ - One data source → one tag; one tag → many data sources.
+
+ #### Relationships (Conceptually)
+
+ | Relationship | Type | Description |
+ |---------------|------|-------------|
+ | Employee ↔ Message | 1 → N | One employee can send multiple messages |
+ | Employee ↔ Tag (via employee_tag) | M ↔ N | Employees can belong to multiple tags |
+ | Message ↔ Data Source | 1 → N | Each message belongs to one data source |
+ | Data Source ↔ Tag | 1 → N | One tag can link to multiple data sources |
+ | Tag ↔ Employee_Tag | 1 → N | Each tag can appear in multiple employee_tag records |
+ | Employee ↔ Employee_Tag | 1 → N | Each employee can have multiple tag associations |
+
+ ---
+
+ ### 6.4 AI Workflow Automation Layer
+
+ - **Retriever:** Uses pgvector to fetch top-K relevant messages.
+ - **LLM (OpenAI GPT-4 / LLaMA):**
+ - Performs Q&A generation.
+ - **LangChain:** Used for the RAG (Retrieval-Augmented Generation) pipeline.
+
+ ---
+
+ ### 6.5 Backend (API Layer)
+
+ **Framework:** FastAPI
+
+ #### Key Endpoints
+
+ | Method | Endpoint | Description |
+ |--------|-----------|-------------|
+ | GET | `/employee/{id}` | Retrieve a specific employee’s profile |
+ | GET | `/employee` | List all employees (Admin only) |
+ | POST | `/message/bulk_ingest` | Store messages fetched from integrations |
+ | POST | `/tag` | Create or update a tag (Admin only) |
+ | GET | `/tag` | Retrieve all active tags |
+ | GET | `/tag/{id}` | Get tag details with linked employees & sources |
+ | POST | `/employee_tag` | Assign or unassign a tag to an employee |
+ | GET | `/employee_tag/{employee_id}` | Fetch tags assigned to a specific employee |
+ | POST | `/data_source` | Add or update a data source |
+ | GET | `/data_source` | List all configured data sources |
+ | GET | `/data_source/{id}` | Get details of a specific data source |
+ | POST | `/chatbot/query` | Ask a natural language question |
+ | GET | `/admin/overview` | Dashboard summary for Admins |
+
+ ---
+
+ ### 6.6 Frontend (React)
+
+ #### Employee View
+
+ - **Profile View**
+ - Displays employee details, department, tenure, and assigned tags.
+ - Uses `/employee/{id}` and `/employee_tag/{employee_id}` APIs.
+
+ - **AI Assistant Chat**
+ - Interactive chat that calls `/chatbot/query`.
+ - Allows employees to query messages related to assigned tags.
+
+ #### HR/Manager/Admin View
+
+ - **Profile Page**
+ - Similar to employee view, with added admin controls.
+
+ - **Tag Management Panel**
+ - Create/edit tags via `/tag` API.
+ - Assign tags to employees via `/employee_tag`.
+
+ - **Data Source Management**
+ - Manage Telegram/Google Chat integrations.
+ - Create or deactivate sources using `/data_source`.
+
+ ---
+
+ ## 7. Data Flow Example
+
+ **Example Query:** “Did we use Kubernetes in any project?”
+
+ 1. Chatbot API receives query.
+ 2. NLP extracts keyword → *Kubernetes*.
+ 3. pgvector performs semantic similarity search.
+ 4. Relevant messages + metadata passed into LLM.
+ 5. LLM responds → “Yes, Kubernetes was used in Project Alpha (2024).”
+ 6. Frontend chatbot displays the answer with source link.
+
+ ---
+
+ ## 8. Tech Stack
+
+ | Component | Technology |
+ |------------|-------------|
+ | **Frontend** | React, Tailwind |
+ | **Backend** | FastAPI (Python) |
+ | **Database** | PostgreSQL + pgvector |
+ | **AI/NLP** | OpenAI Embeddings, GPT-4 |
+ | **Integrations** | Google Chat API, Telegram Bot API |
+ | **Infra** | Docker Compose (MVP), AWS/DigitalOcean |
+
+ ---
+
+ ## 9. Security & Compliance (Outside MVP)
+
+ ### 9.1 Data Protection
+ - Data processed locally or within client’s environment.
+ - No raw chat data stored post-extraction.
+ - Encryption: HTTPS (in transit), AES-256 (at rest).
+
+ ### 9.2 Access Control & Authentication
+ - Role-based access: Admin, Manager, Employee.
+ - OAuth 2.0 for integrations (Google, Telegram).
+ - Logs and audit trails for sensitive actions.
+
+ ### 9.3 SOC 2 & GDPR Alignment
+ - Privacy-by-design principles.
+ - Data minimization: only relevant skill/project data extracted.
+ - Right to erasure and consent-based processing.
+ - Future goal: SOC 2 Type 2 & GDPR certification.
+
+ ### 9.4 Data Retention & Anonymization
+ - Temporary data auto-deletion post-processing.
+ - PII replaced with internal IDs before storage.
+ - Configurable retention policies per client.
+
+ ---
+
+ ## 10. Future Enhancements
+
+ - Integration with Slack, Microsoft Teams, Jira, Confluence, etc.
+ - **Employee Handover Package Generator** during role transitions.
+
+ ---
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9