High-Throughput Distributed Vehicle Telemetry Pipeline
AeroTrack is an event-driven telemetry processing system built in Go for ingesting, processing, storing, and analyzing high-volume vehicle telemetry.
The system uses Apache Kafka as the event backbone, TimescaleDB for time-series storage, Redis for low-latency state, rate-limiting and location lookups, and Gemini for intelligent driving alerts.
The pipeline was load-tested at approximately 550 RPS with 100 concurrent virtual users, achieving 5.25 ms p95 ingestion latency while processing 48,000+ telemetry events with 0% data loss.
A telemetry event moves through the system as follows:
- The simulator generates vehicle telemetry such as location, speed, acceleration, and vehicle ID.
- The Go API receives incoming telemetry requests.
- Requests are placed into bounded in-memory channels rather than performing synchronous database writes.
- A worker pool processes the buffered requests and publishes events to Kafka.
- Kafka provides durable, asynchronous communication between ingestion and processing.
- The background processor consumes telemetry events from Kafka.
- Historical telemetry is persisted to TimescaleDB.
- The latest vehicle state is stored in Redis for low-latency access.
- The alert engine evaluates potentially dangerous driving behavior.
- Gemini can provide an AI-based assessment, with automatic fallback to deterministic rule-based evaluation when the external AI service times out.
Synchronous database writes created a bottleneck in the ingestion path.
Instead, AeroTrack uses:
- Bounded channel buffers
- A worker pool
- Asynchronous Kafka publishing
This keeps the HTTP request path lightweight and allows database processing to happen independently.
Apache Kafka decouples telemetry ingestion from downstream processing.
This provides:
- Durable event buffering
- Independent producer and consumer scaling
- Resilience during traffic spikes
- Decoupling between the API and processing layer
The processor can continue consuming events after temporary downstream slowdowns without blocking the ingestion API.
Vehicle telemetry is inherently time-series data, making TimescaleDB a natural fit for historical telemetry storage and time-based queries.
Redis maintains the latest known state of vehicles and provides fast location lookups required by the alert engine.
This avoids repeatedly querying historical telemetry from the database for frequently accessed state.
The alert engine uses Gemini for intelligent driving assessments.
Because external AI services can fail or timeout, AI evaluation is not treated as a single point of failure. External AI calls are also rate-limited using Redis.
If the Gemini request fails or exceeds the timeout threshold, the system automatically falls back to rule-based evaluation.
AeroTrack was tested using k6 with 100 virtual users generating sustained telemetry traffic.
| Metric | Result |
|---|---|
| Concurrent virtual users | 100 |
| Throughput | ~550 RPS |
| p95 latency | 5.25 ms |
| Events processed | 48,000+ |
| Data loss | 0% |
| Consumer lag during peak traffic | 0 |
The asynchronous ingestion architecture achieved approximately 3.7× the throughput of the original synchronous database-write implementation.
The synchronous implementation forced each request to wait for downstream database work.
The refactored pipeline instead:
HTTP Request -> Bounded Channel -> Worker Pool -> Kafka -> Background Processing -> Timescale and Redis
This removes expensive database operations from the critical HTTP request path.
The core services were refactored around dependency injection and Go interfaces.
External dependencies such as:
- Kafka
- Redis
- AI services
are abstracted behind interfaces.
This allows unit tests to replace real infrastructure with mocks.
Tests are organized as table-driven tests and can execute in milliseconds without requiring Docker containers or external services.
This makes the core business logic independently testable from infrastructure.
- Go
- REST APIs
- Goroutines
- Channels
- Worker Pools
- Dependency Injection
- Apache Kafka
- Event-driven architecture
- Asynchronous processing
- TimescaleDB
- PostgreSQL
- Redis
- Google Gemini API
- Docker
- Azure VM
- Go testing
- k6
Make sure you have:
- Docker
- Docker Compose
- Go
- Git
git clone https://github.com/AzmeerX/AeroTrack.git cd AeroTrack
docker compose up -d
This starts the required infrastructure services including Kafka, Redis, and TimescaleDB.
go run ./Server
go run ./Processor
go run ./Simulator
AeroTrack uses k6 for load testing.
Run in the root:
k6 run stress_test.js
The stress test generates concurrent telemetry requests against the ingestion API and measures throughput and latency under load.
{
"vehicle_id": "001",
"timestamp": "2026-08-22T10:00:00Z",
"latitude": 24.8607,
"longitude": 67.0011,
"speed": 82.4
}The event is accepted by the ingestion service and asynchronously propagated through Kafka to the processing pipeline.
The complete pipeline was deployed to an Azure VM using Docker-based services.
The deployment consists of the ingestion API, Kafka-based messaging, background processing, TimescaleDB persistence, Redis caching, and the alert engine.
- Designed an event-driven telemetry pipeline using Go and Apache Kafka.
- Implemented bounded channels and worker pools for asynchronous ingestion.
- Achieved ~550 RPS at 5.25 ms p95 latency under 100 concurrent virtual users.
- Processed 48,000+ events with 0% data loss during load testing.
- Used TimescaleDB for high-volume time-series telemetry.
- Used Redis for low-latency vehicle state and location lookups.
- Designed AI alerts with deterministic rule-based fallback.
- Applied dependency injection and interfaces to isolate infrastructure dependencies.
- Built fast table-driven unit tests without requiring external containers.
- Deployed the complete pipeline to an Azure VM.
Potential next steps include:
- Horizontal scaling of Kafka consumers
- Kafka partitioning based on vehicle ID
- Schema validation and versioning
- Distributed tracing with OpenTelemetry
- Prometheus/Grafana monitoring
- Kubernetes deployment
- Automated CI/CD deployment pipeline