Feature_01 — Distributed Systems
Millions of user events.One decision engine.
- Role
- Architecture, Backend
- Stack
- Java · Spring Boot · Apache Kafka · Redis · Docker · Grafana
- Elsewhere
- Repository ↗Live ↗
How ten independent services were taught to hold a single conversation — without ever speaking to one another directly.
A distributed recommendation platform built as an event-driven organism — ten Spring Boot services speaking asynchronously through Kafka, each one deaf to the others except through the log.
01
Premise
Premise
Recommendation is a latency problem disguised as a relevance problem. RetargetIQ separates the two: retrieval and ranking answer the request, while analytics and the feature store answer the future. Nothing waits on anything it does not need.
02
Method
Method
Ten Spring Boot microservices communicate over Apache Kafka partitions and consumer groups. Redis holds the hot feature layer. Docker Compose makes the entire topology reproducible on a laptop, then unchanged in deployment.
03
Instrumentation
Instrumentation
Prometheus scrapes throughput and per-service latency; Grafana turns the stream into something a human can read at 2am. Observability was written before the features it observes.
“A system is not its diagram. It is what survives the moment a service disappears.”
Architecture
The topology, read top to bottom.
Fig. 01a — topology
Engineering Decisions
3 entries01
The log is the contract
No service calls another by name. Every fact enters as an immutable Kafka record, and every consumer is free to fall behind, replay, or disappear entirely. The topology can be redrawn without renegotiating a single interface.
02
Retrieval and ranking are separate clocks
Retrieval answers in milliseconds from a narrowed candidate set. Ranking is allowed to be slower and smarter. Splitting them meant the request path stopped inheriting the cost of intelligence.
03
Observability written first
Prometheus counters and Grafana boards existed before the features they measure. A system you cannot read at 2am is a system you do not actually operate.
Challenges
What resisted.
Consumer lag under burst
Impression traffic is not uniform; it arrives in waves. Partition keys were rebalanced around user identity so a single hot cohort could not starve an entire consumer group.
Cache truth versus log truth
Redis held the fast answer, Kafka held the correct one. Invalidation was moved onto the event stream itself, making staleness a bounded, measurable quantity instead of a rumour.
Trade-offs
Eventual consistency
Against — Synchronous accuracyA recommendation that is two seconds old is useful. A recommendation that blocks the page is not.
Ten small services
Against — One legible monolithThe cost is operational surface. The return is that failure stays local and deployment stays independent.
0+
Microservices
0
Kafka topics
0%
Containerised
Timeline
Topology
Event taxonomy, topic design, partitioning strategy.
Services
Gateway, retrieval, ranking and feature store brought up in isolation.
Instrumentation
Metrics, dashboards, and failure drills against a live stream.
Containment
Full Docker Compose parity between laptop and deployment.
Gallery
1spring.kafka.consumer.group-id: ranking2spring.kafka.listener.concurrency: 63max.poll.records: 2504partition.assignment.strategy: CooperativeSticky
Fig. 01a — consumer group rebalancing under burst
1featureStore.merge(userId, signals);2ranker.invalidate(userId);3metrics.counter("features.merged").increment();
Fig. 01b — invalidation travels on the stream
Fig. 01c — signal field
Results
Throughput
Sustained ingestion across six topics with bounded consumer lag under synthetic burst load.
Resilience
Any single service can be killed mid-stream; the log replays it back into agreement.
Reproducibility
The entire ten-service topology stands up from one command.