💬🧠📎 Tracey.

Think: Clippy... but for GNW traces. A suite of tools intended to make GNW traces accessible, regardless of technical capability.

ℹ️ What this tool does Tracey allows you quickly pull and explore traces from Langfuse.

• 📥 Fetch a single set of traces once
• 📊 Explore the same dataset across tabs
• 📋 Generate reports & understand user behaviour
• 🧪 Sample for human eval & product mining

Run locally

Prereqs

• Python >=3.11
• Recommended: uv

1) Set environment variables

Create a .env file in the repo root:

LANGFUSE_PUBLIC_KEY="..."
LANGFUSE_SECRET_KEY="..."
LANGFUSE_BASE_URL="..."

# Optional (only needed for Gemini-powered features)
GEMINI_API_KEY="..."  # or GOOGLE_API_KEY

2) Install dependencies

Option A: uv (recommended)

uv sync

Option B: pip

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e .

3) Run the Streamlit app

uv run streamlit run streamlit_app.py

Then open the URL Streamlit prints (usually http://localhost:8501).

Notes

• The app fetches traces once from the sidebar, then reuses the same dataset across tabs.
• Human eval exports are always available via the Download CSV button.

Metrics documentation

Tracey includes a lightweight documentation layer so pages are self-explanatory:

• Inline KPI tooltips (ℹ️) for key metrics on Analytics / Content KPIs / Thread QA / Eval Insights / QA Sample Packs / CodeAct Templates.
• A dedicated 📚 Metrics Glossary page (in the Streamlit sidebar) that lists all registered metrics with:
  • definition
  • computation / formula
  • provenance (raw Langfuse field vs deterministic derivation)
  • caveats
• A per-page "How to read this page" expander explaining the data source + common pitfalls.

Updating / adding metric definitions

All metric and page docs live in:

• utils/metrics_registry.py (single source of truth)

When you add a new KPI to the UI:

1. Add a doc entry to METRICS (pick a stable metric_id).
2. Render the KPI with metric_with_help(..., metric_id="...") from utils/docs_ui.py.

This keeps the glossary + tooltips in sync.

Product Development Mining

The 🧠 Product intelligence tab contains Product Development Mining, split into three sub-tabs:

• Evidence Mining: search for traces that support a hypothesis (LLM-scored relevance).
• Tagging: LLM-as-judge tagging for prompt topics/flavours and other criteria.
• Gap Analysis: generate a markdown report on user jobs, coverage, and gaps.

LLM settings

In that tab, open ⚙️ LLM Settings to configure:

• Gemini model
• Optional batching (Batch traces per Gemini request, batch size, max chars per trace)

Editable prompts

Each sub-tab exposes an 📝 Edit system prompt expander so you can inspect and tweak the prompts used for Gemini.