Brand Data Pipeline

A Python-based data pipeline that collects product and emissions data from multiple sources and combines them into a cohesive, structured output.

Overview

This pipeline:

1. Scrapes product data from Shopify stores (Acure and Saie)

   • Product names and prices

2. Extracts emissions data from PDF reports

   • Scope 1, 2, and 3 emissions measurements
   • Carbon offset information
   • Reduction targets

3. Combines all data into structured tables with a clear schema

Project Structure

Commons_TakeHoem/
├── data/
│   ├── raw/              # Input PDF files
│   └── output/           # Generated output files
├── src/
│   ├── __init__.py
│   ├── product_scraper.py       # Shopify product scraper with Selenium
name normalization
│   ├── pdf_parser.py            # PDF emissions data parser
│   ├── data_schema.py            # Data schema definitions
│   └── data_combiner.py          # Data combination logic
├── main_pipeline.py      # Main pipeline orchestrator
├── requirements.txt      # Python dependencies
└── README.md

Installation

1. Install Python dependencies:

   pip install -r requirements.txt

2. Install OCR dependencies (optional, for scanned PDFs):

   # macOS
   brew install tesseract poppler
   
   # Ubuntu/Debian
   sudo apt-get install tesseract-ocr poppler-utils
   
   # Windows - download from:
   # https://github.com/UB-Mannheim/tesseract/wiki
   # https://poppler.freedesktop.org/

   • OCR is used to extract text from scanned/image-based PDFs
   • If not installed, the parser will still work but may miss data from scanned PDFs

3. Set up OpenAI API key:

   export OPENAI_API_KEY="your-api-key-here"

   • If no API key is set, the normalizer will use manual mappings only

4. Ensure PDF files are in the data/raw/ directory:

   • Acure_2025_ClimateLabel.pdf
   • Saie_2024_ImpactReport.pdf

Usage

Basic Usage

Run the complete pipeline:

python main_pipeline.py

This will:

• Scrape products from Acure and Saie Shopify stores
• Parse emissions data from PDFs in data/raw/
• Combine all data and save to data/output/ as CSV files

Advanced Usage

# Specify custom directories
python main_pipeline.py --pdf-dir data/raw --output-dir data/output

# Choose output format (csv, excel, parquet, or sql)
python main_pipeline.py --format excel

# Display answers to emissions questions
python main_pipeline.py --answer-questions

Output Schema

The pipeline generates three main tables:

1. Products Table

• brand: Brand name
• product_name: Product name
• price: Product price (USD)
• product_url: URL to product page
• product_id: Unique product identifier

2. Emissions Table

• brand: Brand name
• year: Year of the emissions report
• measures_scope1: Boolean flag for Scope 1 measurements
• measures_scope2: Boolean flag for Scope 2 measurements
• measures_scope3: Boolean flag for Scope 3 measurements
• scope1_value: Scope 1 emissions value in tCO2e (float or empty)
• scope2_value: Scope 2 emissions value in tCO2e (float or empty)
• scope3_value: Scope 3 emissions value in tCO2e (float or empty)
• total_ghg: Total GHG emissions in tCO2e when no clear scope separation (float or empty)
• total_emissions: Total emissions in tCO2e (float or empty)
• has_offsets: Boolean flag indicating if brand offsets emissions
• offsets_currency: Currency of offsets (string or None)
• has_reduction_targets: Boolean flag for reduction targets
• reduction_target_percentage: Target reduction percentage (float or empty)
• reduction_target_year: Target year for reduction (int or empty)
• reduction_target_base_year: Base year for reduction target (int or empty)
• carbonneutral_target: Boolean flag indicating if brand has a carbon neutral target
• carbonneutral_year: Year of carbon neutral target (int or empty, must start with 20XX)

3. Products Emissions Table

One row per product with emissions data joined from the emissions table:

• brand_name: Brand name
• product_name: Name of the product
• year: Year of the emissions report
• product_cost: Product price in USD
• measure_scope1: Scope 1 emissions value in tCO2e, or "NA" if not measured
• measure_scope2: Scope 2 emissions value in tCO2e, or "NA" if not measured
• measure_scope3: Scope 3 emissions value in tCO2e, or "NA" if not measured
• total_ghg: Total GHG emissions in tCO2e when no clear scope separation, or "NA"
• offset_emissions: Boolean indicating if brand offsets emissions (true/false)
• carbonneutral_target: Boolean indicating if brand has a carbon neutral target (true/false)
• carbonneutral_year: Year of carbon neutral target, or "NA" if no target

Emissions Questions Answered

The pipeline extracts data to answer:

1. Does the brand measure their emissions (scopes 1, 2, and 3)?

   • Extracted from PDF text analysis
   • Values stored in emissions table

2. Does the brand offset their emissions?

   • Searches for "carbon footprint offset" and similar keywords
   • Stored in has_offsets field (boolean: true/false)

3. Do they have reduction targets? If so, what are those targets?

   • Extracts target percentages, years, and base years
   • Stored in reduction target fields

Adding New Brands

To add a new brand:

1. Add to product scraper:

   # In src/product_scraper.py, add to scrape_all_brands():
   brands = [
       {'url': 'https://newbrand.com', 'name': 'NewBrand'},
       # ... existing brands
   ]

2. Add PDF to data/raw/:

   • Place PDF file in data/raw/ directory
   • Filename should contain brand name (e.g., NewBrand_2024_Report.pdf)

3. Run pipeline:

   python main_pipeline.py

The pipeline will automatically detect and process the new brand.

Output Formats

• CSV: Separate CSV file for each table
• Excel: Single Excel file with multiple sheets
• Parquet: Efficient columnar format for large datasets
• SQL: SQLite database with separate tables

Dependencies

• requests: HTTP requests for web scraping
• beautifulsoup4: HTML parsing
• pandas: Data manipulation and analysis
• pdfplumber: PDF text extraction
• sqlalchemy: Database operations (for SQL output)
• openpyxl: Excel file support

Notes

• Product Scraping:

  • Uses Shopify's public API endpoint (/products.json) which is rate-limited
  • Uses Selenium to visit individual product pages

• PDF Parsing:

  • Uses multiple extraction methods: text extraction, table parsing, OCR (for scanned PDFs), and computer vision API (optional)
  • Extracts Scope 1, 2, and 3 emissions values by finding numbers followed by "CO" (with possible letters/subscripts) near scope keywords
  • If no clear scope separation exists and only one emissions number is found, stores it in total_ghg
  • Extracts carbon neutral targets by looking for "net-zero", "climate pledge", and "carbon neutral" with years starting with 20XX
  • Detects offsetting by searching for "carbon footprint offset" and similar keywords (returns bool)
  • Automatically detects and extracts data from tables
  • Falls back to OCR if text extraction yields insufficient content
  • Optional: Uses OpenAI Vision API for better extraction from complex PDFs (requires OpenAI API key)

• General:

  • The pipeline is designed to be re-runnable and will overwrite previous outputs
  • Selenium runs in headless mode (no browser window)

Troubleshooting

No products scraped:

• Check if Shopify stores are accessible
• Verify store URLs are correct
• Some stores may block automated requests

No emissions data extracted:

• Verify PDF files are in the correct directory
• Check if PDFs contain text (not just images)
• Review raw_text_snippets in emissions table for manual extraction

Import errors:

• Ensure all dependencies are installed: pip install -r requirements.txt
• Check Python version (3.7+ required)

License

This project is for demonstration purposes.