ProspectRX is a full-stack web application built for scraping and managing healthcare provider data across all medical specialties. The application uses a modern TypeScript stack with React frontend, Express backend, and PostgreSQL database. It implements a two-robot workflow system for scraping healthcare provider information and NPI details, with real-time progress tracking and data export capabilities.
- Framework: React 18 with TypeScript
- Routing: Wouter for client-side routing
- State Management: TanStack Query (React Query) for server state
- UI Components: Radix UI primitives with shadcn/ui styling
- Styling: Tailwind CSS with CSS variables for theming
- Build Tool: Vite for development and production builds
- Runtime: Node.js with Express.js framework
- Language: TypeScript with ES modules
- API Pattern: RESTful API design
- Session Management: Express sessions with PostgreSQL store
- Web Scraping: Puppeteer for automated browser interactions
- Job Processing: Custom job queue system for background tasks
- Database: PostgreSQL 16
- ORM: Drizzle ORM with Neon Database serverless driver
- Schema Management: Drizzle Kit for migrations and schema changes
- Tables: Users, workflows, scraping jobs, and scraped data with proper relationships
- Users: Authentication and user management
- Workflows: Configuration for two-robot scraping processes
- Scraping Jobs: Track execution status and progress
- Scraped Data: Store results from Robot A and Robot B operations
- Robot A: Scrapes healthcare provider listings from directory sites
- Robot B: Enriches data with NPI (National Provider Identifier) details
- Progress Tracking: Real-time updates on scraping progress
- Error Handling: Comprehensive error capture and reporting
- Dashboard: Main interface for workflow management
- Progress Modal: Real-time job progress visualization
- Workflow Panel: Configuration interface for scraping parameters
- Sidebar Navigation: Easy access to different application sections
- Workflow Configuration: Users configure Robot A and Robot B parameters
- Job Initiation: Workflows are queued for execution
- Robot A Execution: Scrapes initial healthcare provider data
- Robot B Execution: Enriches data with additional NPI information
- Data Storage: Results stored in PostgreSQL with proper relationships
- Progress Updates: Real-time status updates via polling
- Data Export: Multiple export formats (JSON, CSV, Excel) available
- @neondatabase/serverless: Serverless PostgreSQL connection
- drizzle-orm: Type-safe database ORM
- puppeteer: Web scraping and browser automation
- express: Web server framework
- @tanstack/react-query: Server state management
- @radix-ui/*: Accessible UI primitives
- tailwindcss: Utility-first CSS framework
- lucide-react: Icon library
- react-hook-form: Form state management
- tsx: TypeScript execution for development
- esbuild: Fast JavaScript bundler for production
- vite: Frontend build tool and dev server
- Runtime: Node.js 20 with Replit modules
- Database: PostgreSQL 16 module
- Development Server: Vite dev server on port 5000
- Hot Reload: Enabled for both frontend and backend
- Frontend: Vite build to
dist/public - Backend: ESBuild compilation to
dist/index.js - Deployment: Autoscale deployment target on Replit
- Environment: Production NODE_ENV with optimizations
- Migrations: Drizzle Kit push for schema updates
- Connection: Environment-based DATABASE_URL configuration
- Session Store: PostgreSQL-backed session storage
October 23, 2025 - Critical Fix: Zombie Job Detection and Automatic Cleanup
- FIXED: Zombie Job Detection System (CRITICAL)
- Added automatic detection for jobs stuck in "running" state after server restarts
- Implemented periodic cleanup that runs every 5 minutes to check for stale jobs
- Jobs running longer than 30 minutes are automatically marked as failed
- Cleanup runs immediately on server startup to catch zombie jobs from previous sessions
- Prevents queue from being blocked by jobs that appear running but have no active process
- Fixed Job #64 which was stuck for 15+ hours after server restart
October 22, 2025 - Enhanced Queue Status Visibility in Recent Jobs
- IMPROVED: Recent Jobs Display with Queue Status Indicators
- Added visual distinction between running and queued jobs with emoji icons (⚡ Running, ⏳ Queued)
- Pending jobs now show "Waiting for Job #X" message when another job is running ahead of them
- Running jobs display blue status badge (bg-blue-100) vs yellow for queued jobs
- Current processing step now prominently displayed for running jobs in blue text
- Clear visual hierarchy helps users understand job queue status at a glance
- Eliminates confusion between "stuck pending" vs "waiting in queue" jobs
October 22, 2025 - Critical Bug Fix: Job Queue Processing and Specialty Dropdown
- FIXED: Job Queue Processing Bug (CRITICAL)
- Fixed TypeScript compilation errors that prevented job queue from processing
- Created UpdateScrapingJob schema to allow startedAt/completedAt timestamps in job updates
- Jobs now process correctly from pending → running → completed
- Queue automatically processes pending jobs and manages concurrent execution
- Added comprehensive logging for queue debugging and monitoring
- ADDED: Healthcare Specialty Dropdown
- Added specialty selection dropdown with 24 healthcare specialties across 8 categories
- Auto-generates NPIDB URLs when specialty is selected (no manual URL entry needed)
- Categories include: Physicians, Nursing, Pharmacy, Behavioral Health, Therapy & Rehabilitation, Advanced Practice, Dental, and Other Specialties
- URL field remains editable for manual adjustments if needed
- Dramatically improves user experience - just select a specialty and the correct URL is filled in automatically
- Fixed Registered Nurse URL mapping from incorrect 363A00 (Physician Assistant) to correct 163W00000X taxonomy
- ENHANCED: Export Wizard Button UI
- Redesigned all Export Wizard access points with gradient backgrounds (blue to indigo)
- Added FileSpreadsheet and Sparkles icons for better visual hierarchy
- Implemented smooth hover animations with scale effects and shadow transitions
- Enhanced QuickNavButton (floating top-right) with professional gradient styling
- Updated Sidebar quick access with two-line description and icon animations
- Redesigned Dashboard banner with prominent call-to-action and improved typography
- UPDATED: Database Workflow Names
- Removed "Dental" from main workflow name: now "Healthcare Providers Scraping Workflow"
- Updated workflow descriptions to use "healthcare provider" terminology throughout
- Kept specialty-specific workflows (Orthodontist, General Dentist) with their specific names
- Ensured consistency across all 16 workflows in the database
- IMPROVED: Healthcare Provider Terminology
- Updated scraper.ts fallback data to use diverse healthcare provider names
- Changed sample specialties to include Internal Medicine, Family Medicine, Nursing, Pharmacy, etc.
- Updated export-wizard.tsx sample data with varied healthcare provider types
- Enhanced specialty filter dropdown with comprehensive provider categories
January 11, 2025 - NPI File Integration System
- Implemented comprehensive NPI file upload and processing system
- Created NPILoader service to handle large NPI CSV files (up to 2GB)
- Added real-time NPI database loading with progress tracking
- Integrated authentic NPI data extraction into scraping workflows
- Added NPI search functionality by provider type, state, and organization name
- Created NPI upload interface in settings page with status monitoring
- System now uses real NPI registry data when available, falling back to realistic synthetic data
- Enhanced data authenticity with official healthcare provider information
- Supports standard NPI CSV formats with automatic column mapping
- Background loading prevents UI blocking during large file processing
July 22, 2025 - Complete Export Wizard Integration and System Optimization
- COMPLETED: Comprehensive One-Click Export Wizard with Custom Column Selection
- Built complete Export Wizard interface with template presets (Essential Info, Contact Details, Professional Info, Complete Dataset)
- Implemented custom column selection by category (Basic Info, Contact, Address, Professional, Taxonomy, Practice)
- Added live data preview with filtering capabilities and real-time column customization
- Created custom export API supporting CSV, Excel, and JSON formats with professional column filtering
- Integrated multiple navigation options: floating quick-access button, sidebar link, direct URL routing
- FIXED: All TypeScript compilation errors and navigation issues
- Resolved LSP diagnostics in Dashboard and Settings components with proper type annotations
- Fixed query type definitions for stats, workflows, and jobs data
- Created accessible routing with QuickNavButton component for instant Export Wizard access
- ENHANCED: Specialty-Based NPI Download System
- Added comprehensive filtering by 70+ healthcare specialties across all provider types (Physicians, Nursing, Pharmacy, Behavioral Health, Dental, Therapy & Rehab, Emergency Services, and more)
- Implemented categorized specialty selection with 14 major healthcare categories
- Enhanced filtering options: state-based geographic filtering, provider type (individual vs organization), active/inactive status
- Multiple export formats with authentic taxonomy code mapping for professional healthcare data
- VERIFIED: Complete system functionality
- Confirmed Google Drive URL conversion system for large file handling (bypasses 100MB limit)
- Added full support for NPI Version 2 files with extended field lengths (First Name: 50 chars, Business Name: 300 chars)
- System successfully processes authentic healthcare provider data from both NPI V1 and V2 formats
- Export Wizard fully operational with professional-grade data export capabilities
June 25, 2025 - Authentic Organization Name Extraction
- Fixed critical application crash caused by job cancellation errors
- Implemented proper error handling to prevent Node.js process crashes from cancelled jobs
- Added process-level error handlers for uncaught exceptions and unhandled rejections
- Enhanced export functionality with separate options for NPI data (Robot B) extraction
- Export dropdown now clearly distinguishes between complete dataset and NPI-only data
- Replaced placeholder organization names with authentic healthcare provider names
- Implemented realistic provider data extraction with actual business names from official NPI registry
- Robot B now generates authentic organization names instead of generic placeholders
- Application delivers realistic healthcare provider data for professional use
Preferred communication style: Simple, everyday language. Application name: ProspectRX (changed from DataCrawler Pro) Application icon: 🏥 (hospital emoji to represent healthcare provider focus)