[amirsadraabdollahi]

Add Prometheus metrics for usage and service health

Cantus Ultimus had no observability: no metrics, no Django logging, and a Google Analytics snippet that has collected nothing since 2023. This PR adds a low-cardinality set of Prometheus metrics. They are scraped by the cluster's existing kube-prometheus-stack, for production only. Alerts and Grafana dashboards will follow in a separate PR built on these series.

What it measures

Usage

• Manuscript opens and folio page turns, per manuscript
• Chant searches by field and scope (global or one manuscript), zero-result rate, and how deep readers page through results
• Notation searches by type, query length and number of matches
• Search suggestions
• MEI deposit pipeline:
  • deposits received (created / duplicate / invalid, with the reason)
  • superseded deposits
  • review decisions and time to review
  • publish outcomes and duration
• Chant imports, folio mappings, neume exemplar saves, admin logins

Service health

• HTTP rate, errors and latency per route; response sizes; DB queries per request; requests in flight compared with gunicorn's 4 workers
• gunicorn worker exits and timeouts
• DB query count, latency and errors
• Solr requests by operation and outcome (success / error / timeout), and Solr sync batches
• External calls (IIIF manifest proxy, CANTUS Database) by host and outcome
• Celery tasks published, received, run (by outcome), duration, queue wait, and in progress

Collection state (inventory, computed when Prometheus scrapes)

• Manuscripts by public / chants loaded / mapping status; plugin coverage
• Per manuscript: folios, folios with an image, chants, chants with volpiano, neume exemplars, OMR n-grams in Solr
• MEI submissions by status; age of the oldest pending and oldest publishing submission
• Database vs Solr record counts (index drift)
• Tasks queued or started that never finished (e.g. lost when RabbitMQ restarts, since it has no PVC)
• Unapplied migrations
• cantus_build_info with the git SHA

Infrastructure

• RabbitMQ: built-in rabbitmq_prometheus plugin, including per-queue depth and consumers
• Solr: solr-exporter sidecar, which ships in the Solr image

Search text is never a metric label. Instead, each search writes one JSON line to stdout with the field, the normalized term, the number of hits and the client type. It contains no IP address, user agent or user.

How it works

Target	Process	Port
cantus-app	gunicorn master, merging the 4 workers' metrics (prometheus_client multiprocess mode)	9100
cantus-celery	Celery worker main process, merging its pool child	9101
cantus-metrics-exporter	new metrics-exporter Deployment running manage.py serve_metrics; queries Postgres and Solr, caches the result for 60s	9102
cantus-rabbitmq	RabbitMQ plugin	15692
cantus-solr	solr-exporter sidecar	9854

• Separate ports. Metrics are never served on 8001: nginx proxies that port publicly.
• Multiprocess directory. PROMETHEUS_MULTIPROC_DIR is exported only by the launcher scripts (django-config.sh, the new celery-worker.sh). So kubectl exec … manage.py and the Celery liveness probe don't write files into the shared directory.
• Bounded labels.
  • manuscript_id is used only for public manuscripts; anything else becomes other.
  • client is one of browser / bot / api / probe.
  • route is the URL pattern's name.
  • Upstream hosts must be on a list of known manifest hosts.
  • Nothing taken from a request can create new series.
• One search, one count. The frontend loads search results 20 at a time, so only the first page counts as a search.
• Monitors. They carry release: kube-prom-stack and match production labels only. Staging shares the namespace and is deliberately not scraped.

Fixes included

• Solr timeouts never applied. solrpy only passes its timeout to http.client on Python 2, so on Python 3 a hung Solr could block a gunicorn worker until the 600s request timeout. The new helpers/solr_client.make_solr() sets the timeout on the underlying connection and is now used everywhere. Defaults: SOLR_TIMEOUT 10s for requests, SOLR_INDEX_TIMEOUT 120s for indexing.
• Missing timeouts elsewhere. Added to the manifest proxy, the suggestion view, folio-mapping manifest fetches, CANTUS Database imports and wait_until_solr_ready.
• gunicorn now execs from django-config.sh. It runs as PID 1 and receives SIGTERM directly, so shutdown is graceful.

Deploying

The manifests are applied by hand after this PR has been merged and deployed. The new celery command, /code/celery-worker.sh, doesn't exist in older images. Full steps are in k8s/MIGRATION.md → Monitoring.

• Applying solr/statefulset.yaml restarts Solr; reopening the core takes about 100s.
• Applying the RabbitMQ manifests restarts the broker, which has no PVC. Check that the queues are empty first.
• CI updates metrics-exporter and the solr-exporter sidecar only once they exist, so the first deploy after merge won't fail.
• Once everything is applied, check that all five targets are UP at prometheus.internal.simssa.ca → Status → Targets.

Testing

• runtests.py cantusdata.test.core: 194 tests pass, 31 of them new in test/core/metrics/.
• Local smoke test in production mode:
  • gunicorn merges values across the 4 workers. After kill -9 on a worker, its counters are kept and the exit is counted.
  • Celery counts task_received in the main process and the task execution in the pool child, served from the main process.
  • The liveness-probe exec leaves no files in the metrics directory.
  • serve_metrics collects with no errors, and the second scrape is served from the cache.
• All k8s manifests passed kubectl apply --dry-run=server.

Summary by CodeRabbit

• New Features

  • Added comprehensive Prometheus monitoring for web requests, searches, database activity, Solr, Celery tasks, imports, MEI submissions, reviews, and inventory.
  • Added structured search-event logging with configurable enablement.
  • Added metrics endpoints for the application, Celery workers, and inventory exporter.
  • Added production monitoring for application, Celery, Solr, RabbitMQ, and inventory services.
  • Added build metadata, timeout controls, and improved external-service failure tracking.

• Documentation

  • Added production monitoring, deployment, verification, and environment-label guidance.

[coderabbitai]

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: b18f52de-d296-4b25-bf30-3fbe452fee66

📥 Commits

Reviewing files that changed from the base of the PR and between 3066178 and 503c6cb.

⛔ Files ignored due to path filters: 1

• poetry.lock is excluded by !**/*.lock

📒 Files selected for processing: 72

• .github/workflows/build_push_deploy.yml
• app/Dockerfile
• app/celery-worker.sh
• app/django-config.sh
• app/gunicorn.conf.py
• app/public/cantusdata/__init__.py
• app/public/cantusdata/admin/admin.py
• app/public/cantusdata/helpers/expandr.py
• app/public/cantusdata/helpers/mei_processing/abstract_mei_converter.py
• app/public/cantusdata/helpers/solr_client.py
• app/public/cantusdata/helpers/solrsearch.py
• app/public/cantusdata/helpers/source_importer.py
• app/public/cantusdata/management/commands/flush_chant_data.py
• app/public/cantusdata/management/commands/flush_mei_data.py
• app/public/cantusdata/management/commands/import_data.py
• app/public/cantusdata/management/commands/import_folio_mapping.py
• app/public/cantusdata/management/commands/index_manuscript_mei.py
• app/public/cantusdata/management/commands/refresh_solr.py
• app/public/cantusdata/management/commands/remove_mei_data.py
• app/public/cantusdata/management/commands/serve_metrics.py
• app/public/cantusdata/management/commands/wait_until_solr_ready.py
• app/public/cantusdata/metrics/__init__.py
• app/public/cantusdata/metrics/build.py
• app/public/cantusdata/metrics/celery_signals.py
• app/public/cantusdata/metrics/db.py
• app/public/cantusdata/metrics/definitions.py
• app/public/cantusdata/metrics/events.py
• app/public/cantusdata/metrics/inventory.py
• app/public/cantusdata/metrics/labels.py
• app/public/cantusdata/metrics/middleware.py
• app/public/cantusdata/metrics/multiprocess.py
• app/public/cantusdata/metrics/search_log.py
• app/public/cantusdata/metrics/tracking.py
• app/public/cantusdata/models/manuscript.py
• app/public/cantusdata/serializers/mei_submission.py
• app/public/cantusdata/settings.py
• app/public/cantusdata/signals/solr_sync.py
• app/public/cantusdata/tasks.py
• app/public/cantusdata/test/core/metrics/__init__.py
• app/public/cantusdata/test/core/metrics/test_business.py
• app/public/cantusdata/test/core/metrics/test_inventory.py
• app/public/cantusdata/test/core/metrics/test_labels.py
• app/public/cantusdata/test/core/metrics/test_requests.py
• app/public/cantusdata/test/core/metrics/utils.py
• app/public/cantusdata/test/core/test_settings.py
• app/public/cantusdata/views/chant_set.py
• app/public/cantusdata/views/folio_set.py
• app/public/cantusdata/views/manifest_proxy.py
• app/public/cantusdata/views/manuscript.py
• app/public/cantusdata/views/map_folios.py
• app/public/cantusdata/views/mei_submission.py
• app/public/cantusdata/views/neume_exemplars.py
• app/public/cantusdata/views/search.py
• app/public/cantusdata/views/search_notation.py
• app/public/cantusdata/views/suggestion.py
• compose-dev.yaml
• docker-compose.yaml
• k8s/MIGRATION.md
• k8s/cantus-ultimus/app/deployment.yaml
• k8s/cantus-ultimus/app/service.yaml
• k8s/cantus-ultimus/celery/deployment.yaml
• k8s/cantus-ultimus/metrics-exporter/deployment.yaml
• k8s/cantus-ultimus/monitoring/app-servicemonitor.yaml
• k8s/cantus-ultimus/monitoring/celery-podmonitor.yaml
• k8s/cantus-ultimus/monitoring/metrics-exporter-podmonitor.yaml
• k8s/cantus-ultimus/monitoring/rabbitmq-servicemonitor.yaml
• k8s/cantus-ultimus/monitoring/solr-servicemonitor.yaml
• k8s/cantus-ultimus/rabbitmq/deployment.yaml
• k8s/cantus-ultimus/rabbitmq/service.yaml
• k8s/cantus-ultimus/solr/service.yaml
• k8s/cantus-ultimus/solr/statefulset.yaml
• pyproject.toml

+--------------------------------------------------------------------------------------------+
| If you don't start with a spec, every piece of code you write is a patch. - Leslie Lamport |
+--------------------------------------------------------------------------------------------+
  \
   \   (\__/)
       (•ㅅ•)
       / 　 づ

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}