Building a Custom Inference Stack
While most enterprises rely on hosted API providers for generative AI, Netflix chose to bring large language model serving entirely in-house. The streaming giant integrated LLM inference directly into its existing production machine learning environment, treating massive language models with the same operational rigor as its traditional recommendation algorithms.
Building an internal stack required navigating several complex architectural decisions regarding engine selection, model packaging, and strict output constraint enforcement. Netflix originally built its platform on TensorRT LLM, but the team pivoted to vLLM. Open source engines had successfully closed the performance gap with specialized stacks, and vLLM offered vastly superior debugging transparency for inspecting intermediate states. Because many Netflix researchers were already using vLLM locally, the switch drastically reduced the friction of moving prototypes into production.
Decoupling Schemas for Better Packaging
One of the most critical decisions involved how models are packaged for the Triton Inference Server. Netflix opted against hardcoding explicit input and output tensor schemas into the model artifacts.
Instead, developers package a simple JSON configuration pointing to the model weights and tokenizer. Triton's backend reads this file and generates the I/O schemas dynamically at deployment time. This loose coupling is vital for maintainability. If the backend schema was frozen, any update to the frontend request builder would require a coordinated rewrite of the model packaging code.
To ensure internal adoption, Netflix standardized its frontend on the OpenAI compatible API format. This allows internal developers to quickly migrate experiments from external third party models to internally hosted, fine tuned models without rewriting their client libraries.
Solving the Deployment Overlap Bug
Operating GPU deployments at scale introduced a unique routing challenge. GPU instances take significantly longer to boot than standard CPU services. When Netflix deployed a new model version requiring a schema change, the standard red black deployment strategy failed. Upstream consumers could not update their configurations until the new model was entirely live, resulting in a window where legacy requests hit the newly formatted model, causing immediate failures.
Netflix solved this by maintaining completely independent deployments for every single model version. The older instance continues to serve traffic securely until the consumer finishes switching their configuration, completely eliminating the transition error window.
Constrained Decoding at Scale
Netflix often requires models to output strictly formatted data rather than unstructured text. To achieve this, they push constraint logic directly into the decode loop using custom logits processors.
Initially, this Python based implementation created a massive CPU bottleneck, as the GIL prevented the system from processing multiple requests in parallel. End to end latency degraded rapidly under realistic load. The team resolved this by upgrading to a newer vLLM architecture that processes logits at the batch level, bypassing the Python GIL using multithreaded C++ operations. This optimization flattened processing times regardless of batch size, allowing Netflix to guarantee perfectly formatted model outputs at massive production scale.