feat: Add mezmo_analytics source

[biblicalph]

Adds a new multi output source (mezmo_analytics) which collects internally generated events for routing through pipelines.
The source exposes outputs for usage metrics, usage metrics by annotations, log clusters, log cluster samples and log cluster annotations.
The source will be used to egress the internal events to Pulsar. The direct DB write paths will be deprecated as part of the Timescale shutdown.

Ref: LOG-23218

Summary

Vector configuration

How did you test this PR?

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

[biblicalph]

tokio-stream is no longer optional

[biblicalph]

mostly relocation of the previous store::save_in_loop implementation.
The main changes here are:

1. Create analytic batches for the newly added source and publish them
2. Aggregation continues even if there is not DB connection (ensures data is served to the source)

[darinspivey]

@greptile

[greptile-apps]

Greptile Summary

The PR adds a multi-output source that exports internally generated usage and log-clustering analytics for downstream routing.

• Introduces process-global broadcast channels and five named analytics outputs.
• Publishes usage and annotation metrics through the new source.
• Refactors log-clustering aggregation to publish cluster, sample, and usage records while retaining database storage.

Confidence Score: 3/5

The PR does not appear safe to merge because multiple configured sources duplicate analytics records and sustained output backpressure permanently drops batches.

Every source instance still subscribes to the same process-global broadcasts, while lagged receivers continue to discard overwritten batches after a named output remains backpressured.

Files Needing Attention: src/sources/mezmo_analytics/config.rs, lib/vector-core/src/mezmo/analytics.rs

Important Files Changed

Filename	Overview
lib/vector-core/src/mezmo/analytics.rs	Adds global per-output broadcast channels and subscriptions; lagged receivers still permanently lose unread analytics batches.
src/sources/mezmo_analytics/config.rs	Registers the source and all five outputs, but every configured instance still subscribes to the same global event streams.
src/sources/mezmo_analytics/mod.rs	Concurrently forwards analytics batches to their named source outputs.
lib/vector-core/src/usage_metrics/mod.rs	Converts aggregated billing and annotation usage into analytics log events before existing flushes.
src/transforms/mezmo_log_clustering/aggregate.rs	Aggregates log-clustering data and emits cluster, sample, and usage batches alongside database persistence.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    UM[Usage metrics aggregation] --> PUB[Global analytics publishers]
    LC[Log clustering aggregation] --> PUB
    PUB --> U[usage_metrics]
    PUB --> A[usage_metrics_by_annotations]
    PUB --> C[log_clusters]
    PUB --> S[log_cluster_samples]
    PUB --> G[log_cluster_usage]
    U --> MA[mezmo_analytics source]
    A --> MA
    C --> MA
    S --> MA
    G --> MA
    MA --> DS[Downstream pipelines and Pulsar]

Loading

_{Reviews (2): Last reviewed commit: "chore: Emit standard vector internal eve..." | Re-trigger Greptile}