Why batch wasn't enough
Traditional service topology systems use batch processing, aggregating data hourly or daily, then storing complete snapshots. The approach works at a modest scale. It fails at Netflix scale for a simple reason. By the time you see the data, it is already old. During a production incident at 3am, an hour-old dependency map is archaeology, not observability.
The first version of the streaming topology system worked perfectly in the local environment. Production was a different story. Kafka consumers fell behind. Instances ran out of memory. Some nodes received 100x the traffic of others. Garbage collection pauses consumed more CPU than actual business logic. The team had to rebuild almost every component.
The streaming-first architecture
The decision was to build streaming-first. Flow records from multi-region Kafka streams and IPC metrics as Server-Sent Events come in continuously. A reactive pipeline with backpressure processes them. Topology updates land in near real time, typically within tens of minutes, compared to the hours-old or day-old data that batch processing approaches provide.
The freshness wasn't just a nice-to-have. Live events need current data. Incident response needs current data. Change validation needs to see the immediate impact of a deploy. The architecture had to support continuous updates at scale, and the only way to do that was to throw out the batch model entirely.
Backpressure: the hard part
The streaming approach created new problems, and the biggest was the question every streaming system eventually faces: how do you process millions of flow records per second in real time without losing data when downstream systems slow down? The team walked through every backpressure strategy. Buffer everything and hope: simple, dangerous. Drop records: simple, but loses the data the topology map is supposed to surface. Sample: works for some metrics, but for topology, sampling creates phantom dependencies and missing edges. The team ended up with a layered strategy that combines flow control at the consumer, adaptive sampling on the highest-volume streams, and a buffering layer that can absorb short bursts without losing data.
The traffic skew problem
The instance imbalance was the most painful production scar. Some nodes received 100x the traffic of others. The cause was the partitioning of flow records by service, which created hot partitions for the highest-traffic services. The fix was a custom partitioner that spreads hot keys across more consumers, with a small per-key ordering guarantee that doesn't break the topology graph's correctness. The team also had to revisit the consumer rebalance protocol to make sure the new partitioner didn't make rebalances slow enough to cause data loss.
Time-travel queries
The most useful feature of the system, in retrospect, is the ability to ask the topology graph about its own past. A user can pick a point in time, and the system reconstructs the dependency map as it was at that moment. The use case is incident postmortems. You want to know which services were calling which at 3:14am, not which services are calling which now. The reconstruction works because the raw flow records are stored in a time-indexed store, and the aggregation pipeline is deterministic. Both choices were made up front because the team had lived through too many postmortems where the answer to "what was the system doing then?" was "we don't know, the data is gone."
The methodology
The optimization methodology that guided the work was the boring one. Profile. Fix the top bottleneck. Profile again. Fix the next one. Repeat. The team did not try to predict where the next problem would be, because the prediction would have been wrong, and the work to address a wrong prediction was wasted. Each round of profiling and fixing took one to three weeks, and the wins compounded. By the time the system was in production, it was processing millions of flow records per second with sub-second query latency, and the team could answer time-travel questions about the topology graph for any point in the recent past.
The lessons generalize. Most distributed systems that fail at scale fail for the same reason: someone assumed the workload was uniform. It never is. Build for skew, profile aggressively, and assume the next bottleneck is the one you haven't thought of yet.