Skip to content

setup.md

Latest commit

 

History

History
234 lines (161 loc) · 5.81 KB

setup.md

File metadata and controls

234 lines (161 loc) · 5.81 KB
Raw

Doc2MD

Setup Guide

Complete installation instructions for the DocumentToMarkdown pipelines (poster, patent, talk, presentation, paper, CI, outlook).

Step 1: Install Miniconda

Download

  1. Go to https://docs.conda.io/en/latest/miniconda.html
  2. Download Miniconda3 Windows 64-bit
  3. Run installer

Installation

  • Install for: Just Me (recommended)
  • Location: Default (C:\Users\YourName\miniconda3)
  • Add to PATH: ☑ (optional but recommended)

Verify

conda --version
# Should output: conda 23.x.x or newer

Step 2: Create Conda Environment

# Create environment with Python 3.11
conda create -n ds_env python=3.11 -y

# Activate
conda activate ds_env

# Verify
python --version
# Should output: Python 3.11.x

Step 3: Install Packages

conda activate ds_env

# PDF processing
conda install -c conda-forge pdfplumber pymupdf pillow -y

# Data handling
conda install pandas openpyxl -y

# OCR support (used by poster and talk pipelines)
conda install -c conda-forge tesseract pytesseract -y

# Presentation pipeline (PPTX support)
pip install python-pptx

# Outlook pipeline (DOCX attachment extraction + contact YAML files)
pip install python-docx
pip install pyyaml

# Anthropic API (Claude Vision)
pip install anthropic

Verify Installation

python -c "import pdfplumber, fitz, pandas, openpyxl, anthropic, pptx, docx, yaml; print('All packages OK')"

tesseract --version
# Should output: tesseract 5.x.x

Or use the provided environment check script:

python scripts/check_environment.py

Step 4: Configure API Credentials

All three pipelines use the Anthropic API (Claude) for vision analysis. Set credentials as Windows User environment variables:

# Set authentication token (PowerShell)
[Environment]::SetEnvironmentVariable("ANTHROPIC_AUTH_TOKEN", "your-token", "User")

# Set base URL (e.g. corporate proxy)
[Environment]::SetEnvironmentVariable("ANTHROPIC_BASE_URL", "https://your-proxy-url/api/proxy/anthropic", "User")

# Verify
[Environment]::GetEnvironmentVariable('ANTHROPIC_AUTH_TOKEN', 'User')
[Environment]::GetEnvironmentVariable('ANTHROPIC_BASE_URL', 'User')

IMPORTANT: Restart your terminal after setting variables.

Alternatively, if using the Anthropic API directly (no proxy), set ANTHROPIC_API_KEY instead:

[Environment]::SetEnvironmentVariable("ANTHROPIC_API_KEY", "sk-ant-...", "User")

Step 5: Run a Pipeline

# Navigate to project
cd DocumentToMarkdown

# Activate environment
conda activate ds_env

# Process posters
python poster_pipeline.py --sharepoint "path/to/poster_pdfs" --metadata "abstracts.xlsx"

# Process patents
python patent_pipeline.py --input "path/to/patent_pdfs"

# Process talks
python talk_pipeline.py --talks "path/to/talk_pdfs" --metadata "abstracts.xlsx"

# Process presentations
python presentation_pipeline.py --input "path/to/pptx_or_pdf_files"

# Process scientific papers
python paper_pipeline.py --input "path/to/papers"

# Process competitive intelligence documents
python ci_pipeline.py --input "path/to/ci_docs"

# Process Outlook emails (Outlook must be running)
python outlook_pipeline.py --folder "Inbox/CI Reports"

See the README for full CLI options, or the pipeline-specific guides for architecture details:

Troubleshooting

Conda Not Found

Restart terminal or add to PATH manually:

set PATH=%PATH%;C:\Users\YourName\miniconda3\Scripts

Package Import Errors

# Verify environment is active
conda activate ds_env
where python
# Should show: C:\Users\YourName\miniconda3\envs\ds_env\python.exe

# Reinstall a package
conda install -c conda-forge <package-name> -y

Tesseract Not Found

# Add to PATH
[Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\Program Files\Tesseract-OCR", "User")

# Restart terminal

If Tesseract language data is missing, run the automated setup:

python scripts/setup_tessdata.py

API Credentials Not Working

# Check current values
[Environment]::GetEnvironmentVariable('ANTHROPIC_AUTH_TOKEN', 'User')

# Re-set if empty
[Environment]::SetEnvironmentVariable("ANTHROPIC_AUTH_TOKEN", "your-token", "User")
[Environment]::SetEnvironmentVariable("ANTHROPIC_BASE_URL", "https://your-proxy-url/api/proxy/anthropic", "User")

# MUST restart terminal after changes

Installation Checklist

  • Install Miniconda
  • Create ds_env environment (Python 3.11)
  • Install pdfplumber, pymupdf, pandas, openpyxl, anthropic, pytesseract, python-pptx, python-docx, pyyaml
  • Set ANTHROPIC_AUTH_TOKEN and ANTHROPIC_BASE_URL (or ANTHROPIC_API_KEY)
  • Restart terminal
  • Run a pipeline to verify everything works

Package Reference

Package Purpose Install Command
pdfplumber Native PDF text extraction conda install -c conda-forge pdfplumber
pymupdf (fitz) PDF rendering to images conda install -c conda-forge pymupdf
pillow Image processing conda install pillow
pandas Metadata handling conda install pandas
openpyxl Excel file support conda install openpyxl
python-pptx PPTX text/table extraction pip install python-pptx
python-docx DOCX text extraction (Outlook attachments) pip install python-docx
pyyaml YAML read/write for Outlook contact files pip install pyyaml
pytesseract OCR interface (Tesseract) conda install -c conda-forge pytesseract
anthropic Claude Vision API client pip install anthropic