Skip to content

Use xxHash64 for event_instance_id to prevent hash collisions - #85

Merged
tombonfert merged 3 commits into
mainfrom
fix/event_instance_id_collisions
Aug 21, 2026
Merged

Use xxHash64 for event_instance_id to prevent hash collisions#85
tombonfert merged 3 commits into
mainfrom
fix/event_instance_id_collisions

Conversation

@tombonfert

Copy link
Copy Markdown
Collaborator

Problem

event_instance_id was generated with Spark's crc32, whose 32-bit output space produces collisions in practice. A user hit a real case where two distinct instances of the same event on one container (same event_name, different time intervals) hashed to the same id.

Because event_instance_id is a MERGE key for event_instance_fact and the join key from stats_aggregator_fact and point_value_fact, a collision silently drops or mis-associates event instances in the gold layer.

Fix

Switch event_instance_id generation from crc32 to xxhash64 (64-bit), widening the key space enough to keep it collision-free at scale. The change is made in the single shared helper (generate_event_instance_id_column), so event_instance_fact and the aggregation facts stay in lockstep automatically.

Changes

  • event_instance_util.py: crc32 to xxhash64 for both the container-event and timestamp-based branches; docstring updated.
  • fact_schema.py: event_instance_id typed as LongType to match the 64-bit hash and STATS_AGGREGATOR_FACT_SCHEMA (cosmetic; persistence projects by name).
  • Tests and a docstring in stats_aggregator.py updated to mirror the new hash.

Notes

  • event_instance_id values are now 64-bit and can be negative. Equality still holds for MERGE and joins, so correctness is unaffected, but any downstream consumer assuming a positive or 32-bit id should be aware.
  • No gold-table migration. Existing rows keep their old ids; affected instances are re-inserted with new ids on the next run.

Verification

Targeted unit tests, the events/aggregations/incremental sweep, and the statistics integration tests pass; make lint clean.

…t schema to LongType

Replaces the CRC32-based event_instance_id generation with xxHash64 to reduce collisions at scale, and updates `event_instance_id` in `EVENT_INSTANCE_FACT_SCHEMA` from `IntegerType` to `LongType` to hold the signed 64-bit hash values. ContainerEvent rows continue to hash only `container_id`, while other event types hash `container_id::event_name::start_ts::end_ts`.
@tombonfert
tombonfert requested a review from a team as a code owner August 21, 2026 10:04
@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.01%. Comparing base (23541ff) to head (425fa9e).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #85   +/-   ##
=======================================
  Coverage   89.01%   89.01%           
=======================================
  Files          61       61           
  Lines        5207     5207           
  Branches      628      628           
=======================================
  Hits         4635     4635           
  Misses        461      461           
  Partials      111      111           
Flag Coverage Δ
query_engine 84.77% <ø> (ø)
reporting 94.24% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...impulse_reporting/aggregations/stats_aggregator.py 94.25% <ø> (ø)
src/impulse_reporting/persist/fact_schema.py 100.00% <ø> (ø)
src/impulse_reporting/util/event_instance_util.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Replaces the non-negative CRC32 assertion with checks that every BasicEvent instance has an event_instance_id and that all instance ids are distinct, matching the switch to per-instance timestamp-based xxHash64 hashing.
@tombonfert
tombonfert merged commit ce230cd into main Aug 21, 2026
6 checks passed
@tombonfert
tombonfert deleted the fix/event_instance_id_collisions branch August 21, 2026 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant