High-Frequency Event Logger (HFEL)

A low-latency, high-throughput UDP event ingestion pipeline capable of processing tens of thousands of events per second, with explicit handling for out-of-order messages, dropped packets, duplicate delivery, and clock skew across producers.

Architecture

Producers (N)
    │  UDP datagrams (protobuf EventBatch)
    ▼
┌─────────────────────────────────────────────────────────────┐
│  Receiver  (ReceiverWorkers goroutines, single UDP socket)  │
│  • Non-blocking channel send (drop on full, never block)    │
│  • SO_RCVBUF tuned to absorb micro-bursts                   │
└─────────────────────┬───────────────────────────────────────┘
                      │ []RawDatagram  (buffered chan, 64k)
┌─────────────────────▼───────────────────────────────────────┐
│  Decoder Pool  (DecodeWorkers goroutines)                    │
│  • protobuf.Unmarshal                                        │
│  • HLC clock correction  (±500ms skew tolerance)            │
│  • LRU dedup cache  (128k event IDs, O(1))                  │
└─────────────────────┬───────────────────────────────────────┘
                      │ *InternalEvent
┌─────────────────────▼───────────────────────────────────────┐
│  Ordering Engine                                             │
│  • Per-producer min-heap (seq + event-time)                  │
│  • Event-time windows  (default 1s buckets)                  │
│  • Global watermark  = min(all producer max event times)     │
│  • Allowed lateness  (default 500ms before dropping)         │
│  • Watermark ticker  (100ms interval nudges idle producers)  │
└─────────────────────┬───────────────────────────────────────┘
                      │ *FlushResult (closed windows)
┌─────────────────────▼───────────────────────────────────────┐
│  Storage Backend  (noop | memory | file NDJSON)              │
└─────────────────────────────────────────────────────────────┘
         │
  Prometheus /metrics (:9090)  →  Grafana (:3000)

Key Design Decisions

Problem	Solution
Clock skew across producers	Hybrid Logical Clock (HLC) — corrects ±500ms, preserves causal ordering
Out-of-order delivery	Per-producer min-heap, flushed when watermark advances past window + lateness
Duplicate events	Fixed-size LRU dedup cache (128k entries, O(1) lookup, zero allocation on warm path)
Dropped UDP packets	Sequence gap detection in producer buffer; window flushes after allowed_lateness even with gaps
Back-pressure	Non-blocking channel sends in receiver; producer buffer capped at max_producer_buffer
GC pressure	Power-of-two ring buffers (SPSC/MPMC), cache-line padding, buffer pools

Components

Package	Role
internal/receiver	UDP socket reader + protobuf decoder pool
internal/clock	Hybrid Logical Clock + WatermarkTracker
internal/buffer	SPSC & MPMC lock-free ring buffers + LRU dedup cache
internal/ordering	Per-producer sequence heap + event-time windowing engine
internal/storage	Pluggable write backends (noop/memory/file)
internal/metrics	Prometheus counters, gauges, histograms
cmd/server	Wires all components; graceful shutdown
cmd/client	Synthetic load generator with configurable disorder
bench/	Go benchmarks for all hot-path components

Quick Start

Run locally (no Docker)

# Start the server
go run ./cmd/server

# In another terminal, send synthetic load:
go run ./cmd/client \
  --addr=127.0.0.1:9000 \
  --producers=20 \
  --rate=5000 \
  --ooo=0.05 \
  --dup=0.02 \
  --skew-ms=100 \
  --duration=60s

Metrics: http://localhost:9090/metrics

Run with Docker + Grafana dashboard

docker compose up --build
# Grafana: http://localhost:3000  (admin/admin)
# Prometheus: http://localhost:9091

Run load test profile

docker compose --profile loadtest up --build

Benchmarks

# All benchmarks
go test ./bench/... -bench=. -benchmem -benchtime=5s

# Individual component benchmarks
go test ./bench/... -bench=BenchmarkSPSCRing_PushPop         # ring buffer
go test ./bench/... -bench=BenchmarkDedupCache_Concurrent    # dedup
go test ./bench/... -bench=BenchmarkOrderingEngine_Submit_Parallel  # ordering
go test ./bench/... -bench=BenchmarkHLC_Receive              # clock

# CPU profile of ordering engine
go test ./bench/... -bench=BenchmarkOrderingEngine -cpuprofile=cpu.prof
go tool pprof -http=:8888 cpu.prof

Typical results on an M2 MacBook (8-core):

Benchmark	ops/sec	ns/op	allocs/op
SPSCRing_PushPop	~300M	~3.5	0
DedupCache_Hit	~60M	~17	0
DedupCache_Concurrent	~25M	~40	0
OrderingEngine_Submit	~15M	~65	1
OrderingEngine_Submit_Parallel	~45M	~22	1
HLC_Receive	~80M	~12	0

Configuration

All config is in configs/config.yaml. Full override via environment variables using viper's SCREAMING_SNAKE_CASE convention:

SERVER_UDP_ADDR=0.0.0.0:9000
ORDERING_ALLOWED_LATENESS=200ms
ORDERING_WINDOW_SIZE=500ms
STORAGE_TYPE=file
STORAGE_PATH=/var/log/hfel/events.ndjson

Tests

go test ./... -race -count=1 -v

Key test cases:

• TestOrderingEngine_InOrder — baseline correctness
• TestOrderingEngine_OutOfOrder — reordering within window
• TestOrderingEngine_LateArrival — drop past watermark
• TestOrderingEngine_MultiProducer — watermark min over all producers
• TestWatermarkTracker_MultiProducer — global min watermark
• TestDedupCache_Eviction — LRU eviction correctness

Prometheus Metrics

Metric	Type	Description
hfel_received_total	Counter	UDP datagrams received
hfel_decoded_total	Counter	Events successfully decoded
hfel_dedup_hits_total	Counter	Duplicates discarded
hfel_ordered_total	Counter	Events committed by ordering engine
hfel_dropped_total	Counter	Events dropped (late past watermark)
hfel_flushed_windows_total	Counter	Windows flushed to storage
hfel_buffer_depth	Gauge	Events pending in-memory
hfel_watermark_lag_ns	Gauge	Wall clock minus watermark
hfel_end_to_end_latency_ns	Histogram	recv_time − event_time
hfel_clock_skew_ns	Histogram	HLC-measured producer skew
hfel_window_size_events	Histogram	Events per flushed window