The Need for Real-Time Visibility
When a production incident strikes at three in the morning, an hour-old system dependency map is essentially useless. Engineers at Netflix required a unified, real-time view of their vast microservice architecture to accurately gauge blast radius and troubleshoot outages instantly.
Traditional service topology tools rely on batch processing, aggregating historical data daily. Netflix chose to build a streaming-first architecture. The system ingests eBPF network flows, internal process metrics, and distributed tracing data continuously. By utilizing a reactive pipeline equipped with strict backpressure handling, the topology map updates in tens of minutes rather than hours. When downstream graph databases slow down, the backpressure mechanism signals the Kafka consumers to pause, ensuring data is delayed gracefully rather than completely dropped due to overflowing memory buffers.
Resolving the Network Proxy Problem
The primary challenge of processing raw eBPF flow logs is that they only reveal individual network hops. If an application talks to a database through a load balancer, the logs show two separate connections. To be useful for incident response, the topology map must resolve these intermediaries and draw a direct line between the source and the ultimate destination.
Solving this required grouping all flows associated with a specific intermediary proxy on a single physical instance to perform a join operation. However, implementing this logic exposed a catastrophic scaling flaw in the initial pipeline design.
Surviving the Hot Node Crisis
The engineering team originally used a single stage architecture driven by consistent hashing to group the data. This design failed spectacularly in production. Popular internal services, like Netflix's authentication layer, generated orders of magnitude more traffic than standard services.
The instances assigned to process these popular intermediaries became severely overloaded "hot nodes." The data volume multiplied during the network shuffle, causing massive memory spikes. The Java Virtual Machine spent more compute cycles executing garbage collection pauses than executing actual business logic, eventually crashing the instances and triggering cascading failures across the cluster.
The Three-Stage Pipeline Solution
To eliminate the hot nodes, Netflix completely redesigned the ingestion architecture into a three-stage distributed pipeline, swapping heavyweight gRPC calls for lightweight Server Sent Events.
In the first stage, raw network flows are aggregated locally into five minute time windows before ever crossing the network. This immediate local aggregation prevents raw flow objects from piling up and crushing the garbage collector.
The compressed aggregators are then streamed to a second stage, which handles the complex proxy resolution and joins the network edges. Finally, the resolved data is distributed again to a third stage, which queries external datastores for application health metadata before batching the writes into the graph database. By forcing the data through multiple, graduated redistribution steps, Netflix successfully diffused the extreme traffic spikes, ensuring no single instance ever becomes a systemic bottleneck.