Introduction

Welcome to Chapter 3 of our deep dive into Netflix’s internal workings! In the previous chapter, we laid the groundwork by understanding Netflix’s microservices architecture and the principles driving its distributed design. Now, we shift our focus to the very foundation of its global reach and incredible performance: its hybrid infrastructure.

This chapter will explain how Netflix leverages a powerful combination of Amazon Web Services (AWS) for its vast array of backend services and a custom-built Content Delivery Network (CDN) called Open Connect for delivering video streams. Understanding this dual-pronged approach is crucial for grasping how Netflix achieves its unparalleled scalability, resilience, and low-latency streaming experience across over 190 countries.

By the end of this chapter, you’ll have a clear mental model of how a streaming request flows through this distributed global infrastructure, the strategic reasons behind Netflix’s design choices, and the significant tradeoffs involved in operating such a complex system. A fundamental understanding of cloud computing and CDN concepts will be beneficial as we explore these architectural marvels.

System Breakdown: Global Infrastructure

Netflix operates a truly global service, meaning its infrastructure must be distributed worldwide to serve users efficiently. The core architectural decision that underpins this is a hybrid model:

  1. AWS for the Control Plane: All of Netflix’s microservices, databases, data analytics, transcoding, and operational tools run on AWS. This forms the “brain” and “nervous system” of Netflix.
  2. Open Connect Network (OCN) for the Data Plane: Netflix’s proprietary, custom-built CDN is responsible for caching and delivering the actual video content directly to users. This forms the “muscle” that delivers the bits.

This separation of concerns is a foundational aspect of Netflix’s infrastructure design.

AWS: The Global Control Plane

Since migrating off its own data centers in 2008 and completing its cloud migration by 2016, Netflix has extensively utilized AWS for almost all of its non-content delivery computational needs. This strategic choice provides Netflix with immense elasticity, global reach, and access to a vast ecosystem of managed services.

What lives on AWS:

  • Microservices: Thousands of interconnected services that handle everything from user authentication, account management, billing, search, recommendations, social features, and API gateways.
  • Databases: A variety of data stores including NoSQL databases (e.g., Cassandra, DynamoDB), relational databases (e.g., RDS PostgreSQL, MySQL), and specialized data stores for specific service needs.
  • Content Transcoding & Processing: The compute-intensive process of converting raw video files into hundreds of different formats, resolutions, and bitrates suitable for various devices and network conditions.
  • Big Data & Analytics: Platforms for processing massive amounts of user interaction data, content metadata, and operational telemetry to power recommendations, personalization, and business intelligence.
  • Monitoring & Operations: Tools for observing the health, performance, and security of the entire system, critical for Site Reliability Engineering (SRE).
  • CI/CD Pipeline: Tools and services that automate the build, test, and deployment of new software features across thousands of microservices.

Key AWS Services Utilized (Likely/Inferred):

Netflix has publicly detailed its heavy reliance on AWS, though specific service versions or configurations are internal. Common AWS services supporting such an architecture would include:

  • Compute: Amazon EC2 (for custom application servers, often orchestrated by custom schedulers or Kubernetes), AWS Lambda (for event-driven functions).
  • Storage: Amazon S3 (for raw content assets, transcoded video master copies, backups, data lake), Amazon EBS (for EC2 instance storage).
  • Databases: Amazon DynamoDB (for high-scale key-value storage), Amazon RDS (for relational data), Amazon Cassandra (likely custom managed, though AWS Keyspaces is an option now), Elasticsearch for search.
  • Networking: Amazon VPC (virtual private clouds), Amazon Route 53 (DNS), Elastic Load Balancers (ALB/NLB).
  • Messaging/Eventing: Amazon SQS (message queues), Apache Kafka (likely self-managed clusters on EC2 or via a managed service like MSK) for high-throughput event streaming.
  • Security: AWS IAM, KMS, WAF.
  • Monitoring: Amazon CloudWatch, along with Netflix’s extensive custom monitoring tools like Atlas and Spectator.

Open Connect Network (OCN): The Global Data Plane

While the “brain” of Netflix runs on AWS, the “muscles” that deliver billions of hours of video directly to viewers belong to the Open Connect Network (OCN). This is a custom, purpose-built Content Delivery Network designed by Netflix.

Purpose: OCN’s primary goal is to deliver video content with the highest possible quality and lowest latency to Netflix subscribers worldwide, while simultaneously optimizing bandwidth costs for Netflix and its Internet Service Provider (ISP) partners.

Components and Strategy (Fact, per Netflix Open Connect website):

  • Open Connect Appliances (OCAs): These are custom-designed, high-performance servers that Netflix deploys. OCAs contain massive amounts of storage and networking capacity.
  • Strategic Placement: OCAs are strategically deployed in two primary locations globally:
    1. Internet Exchange Points (IXPs): These are physical locations where different internet networks (like ISPs) meet to exchange traffic.
    2. ISP Data Centers: Netflix offers ISPs the option to host OCAs directly within their own data centers.
  • Content Caching: OCAs pre-populate and cache Netflix’s vast content library. Popular content is mirrored across many OCAs, while less popular content might be on fewer. This brings the video bits physically closer to the end-users.
  • Direct Peering: By placing OCAs at IXPs or directly within ISPs, Netflix establishes direct peering relationships. This means video traffic travels fewer network hops, avoids congested transit networks, and results in a faster, more reliable streaming experience.
  • Traffic Management: Netflix maintains sophisticated logic to direct a user’s video request to the closest and least congested OCA, ensuring optimal delivery.

The fact that Netflix provides these OCAs to ISPs for free (along with technical support) is a key aspect of its strategy. This incentivizes ISPs to partner with Netflix, reduces the ISP’s own bandwidth costs for Netflix traffic, and ultimately improves the streaming experience for their mutual customers.

Content Ingestion and Distribution Workflow (Likely Inference & Public Information)

The journey of a movie or TV show from production house to your screen is a multi-stage process involving both AWS and OCN.

  1. Ingestion (AWS): Original, high-quality master video files are uploaded to Netflix, typically stored in long-term archival storage on AWS S3.
  2. Transcoding (AWS): These master files are then fed into Netflix’s sophisticated transcoding pipelines running within AWS. This process converts the single master file into hundreds of different encoded versions (different resolutions, bitrates, audio tracks, subtitles) optimized for various devices (smart TVs, phones, tablets) and network conditions (from low-bandwidth mobile to 4K HDR). This is a highly parallel and compute-intensive operation.
  3. Storage of Masters (AWS): The transcoded “master” files (many versions of the same show) are stored durably, often back in AWS S3 buckets, awaiting distribution.
  4. Distribution to OCN (AWS to OCN - Inference): Based on anticipated demand, new releases, and geographic popularity, these transcoded assets are then distributed from AWS S3 to the geographically dispersed OCAs across the Open Connect Network. This distribution often happens proactively, ahead of user requests, and during off-peak hours to minimize network congestion. Robust internal systems manage the synchronization and integrity of content across thousands of OCAs globally.

How This Part Likely Works: Streaming Request Flow

Let’s trace a typical user request to stream a video, highlighting the interplay between the AWS control plane and the Open Connect data plane.

flowchart TD User_Device[User Device - Netflix App] AWS_Auth_Service["AWS: Authentication Service"] AWS_Playback_API["AWS: Playback API Service"] AWS_Catalog_DB["AWS: Content Catalog DB"] AWS_Manifest_Gen["AWS: Manifest Generation Service"] OCN_DNS_Resolver["OCN: Smart DNS Resolver"] OCN_OCA_Cache["OCN: Open Connect Appliance Cache"] subgraph AWS_Control_Plane["AWS Control Plane "] direction LR AWS_Auth_Service AWS_Playback_API AWS_Catalog_DB AWS_Manifest_Gen end subgraph OCN_Data_Plane["Open Connect Network "] direction LR OCN_DNS_Resolver OCN_OCA_Cache end User_Device -->|\1| AWS_Auth_Service AWS_Auth_Service -->|\1| User_Device User_Device -->|\1| AWS_Playback_API AWS_Playback_API -->|\1| AWS_Catalog_DB AWS_Playback_API -->|\1| AWS_Manifest_Gen AWS_Manifest_Gen -->|\1| AWS_Playback_API AWS_Playback_API -->|\1| User_Device User_Device -->|\1| OCN_DNS_Resolver OCN_DNS_Resolver -->|\1| User_Device User_Device -->|\1| OCN_OCA_Cache OCN_OCA_Cache -->|\1| User_Device

Step-by-Step Flow:

  1. User Interaction with App (AWS Control Plane): When a user launches the Netflix app, it communicates with various microservices hosted on AWS. Initial requests might involve authentication, fetching personalized recommendations, and displaying the content catalog.
  2. Playback Request (AWS Control Plane): When a user selects a video and presses “Play,” the Netflix app sends a request to an AWS-hosted Playback API Service. This service is responsible for orchestrating the start of the stream.
  3. Orchestration and Entitlement (AWS Control Plane): The Playback API Service interacts with other AWS services:
    • It verifies the user’s subscription and regional content rights by querying Authentication & Authorization services and Content Catalog databases.
    • It determines the optimal video format, resolution, and bitrate for the user’s device and current network conditions. This might involve A/B testing or dynamic adjustments.
  4. Manifest Generation (AWS Control Plane): A Manifest Generation service (also on AWS) compiles a “playback manifest” (e.g., an MPEG-DASH or HLS manifest). This manifest is essentially a playlist containing a list of URLs for all the tiny video segments (chunks) that make up the movie or show, for all available bitrates. Crucially, these URLs point directly to the Open Connect Network.
  5. Manifest Delivery (AWS Control Plane to Client): The generated manifest is sent back to the user’s Netflix client application by the Playback API Service.
  6. Video Segment Fetching (OCN Data Plane): The Netflix client application then begins fetching video segments by resolving the URLs provided in the manifest.
    • It performs a DNS lookup for the hostname in the OCN URL. Netflix uses its own Smart DNS Resolver within OCN to direct the client to the closest and best-performing Open Connect Appliance (OCA).
    • The client establishes a connection directly with that OCA.
    • The OCA serves the requested video segment from its local cache.
    • If, in a rare scenario, the specific segment isn’t cached (e.g., for extremely unpopular, niche content), the OCA might fetch it from another OCA or, as a last resort, from the master storage in AWS S3. However, OCN is designed for high cache hit rates.
  7. Continuous Streaming: The client continuously fetches segments from the OCA, buffering them and playing them back, providing a seamless streaming experience. The client-side adaptive bitrate logic dynamically requests higher or lower quality segments based on real-time network conditions.

Tradeoffs & Design Choices

Netflix’s hybrid infrastructure isn’t just a technical achievement; it’s a testament to strategic design choices driven by immense scale and specific business needs.

Benefits of the Hybrid Model (AWS + OCN):

  • Cost Optimization: This is perhaps the most significant benefit. While AWS provides elasticity and a wide array of services, egress bandwidth costs for streaming billions of hours of video from a public cloud would be astronomical. By operating OCN, Netflix drastically reduces its transit costs by directly peering with ISPs and localizing content delivery.
  • Performance and User Experience: Delivering content from OCAs physically located within or very close to ISP networks minimizes latency, reduces buffering, and ensures a higher quality of experience, especially for global audiences. The custom nature of OCN allows Netflix to fine-tune delivery algorithms for optimal streaming.
  • Control and Optimization: Building and operating OCN gives Netflix direct control over the entire video delivery pipeline, from caching strategies to network routing. This allows for continuous optimization and innovation specific to video streaming needs, which might not be possible with a generic third-party CDN.
  • Resilience: The clear separation between the AWS control plane and the OCN data plane provides a level of fault isolation. A localized issue in an OCA typically only affects a subset of users or content, while a major AWS regional outage (though rare) might impact service orchestration but not necessarily stop currently streaming videos (until new segments are needed).

Costs and Complexities of the Hybrid Model:

  • Operational Overhead: Running a global, custom CDN is incredibly complex. It requires significant investment in hardware (design, manufacturing, deployment, maintenance of thousands of OCAs), logistics, network engineering expertise, and relationships with hundreds of ISPs worldwide.
  • CapEx Investment: Unlike pure cloud solutions, OCN requires substantial capital expenditure (CapEx) for hardware and infrastructure deployment.
  • Management of Diverse Infrastructures: Maintaining expertise and operational processes for both a highly dynamic, cloud-native AWS environment and a distributed, hardware-centric OCN adds to the operational complexity.
  • Security Boundary Management: While beneficial for isolation, managing security across two distinct infrastructure domains (AWS and OCN) requires meticulous design and ongoing vigilance.

Common Misconceptions

  1. Netflix runs entirely on AWS: This is the most common misconception. While Netflix heavily relies on AWS for its control plane, logic, and processing, the actual video bits streamed to your device typically come from Netflix’s own Open Connect Network, not directly from AWS.
  2. Open Connect is just like any other CDN: While it performs the function of a CDN, Open Connect is not a generic, off-the-shelf third-party CDN service. It’s a custom-designed hardware and software system, purpose-built by Netflix for its specific streaming needs, deployed directly with ISPs and at IXPs, emphasizing direct peering and cost efficiency.
  3. Netflix transcodes video on Open Connect appliances: Transcoding is an extremely compute-intensive process that happens within the elastic compute resources of AWS. Open Connect appliances primarily store and deliver the already transcoded video files. They are optimized for high-throughput I/O and caching, not for heavy processing.

Summary

This chapter has illuminated the crucial role of Netflix’s global infrastructure, revealing a sophisticated hybrid architecture:

  • Netflix leverages Amazon Web Services (AWS) as its global control plane, hosting all microservices, databases, analytics, content processing (transcoding), and operational tooling. This provides elasticity, agility, and access to a vast array of cloud services.
  • For high-volume, low-latency video delivery, Netflix relies on its custom-built Open Connect Network (OCN). This proprietary data plane consists of specialized hardware appliances strategically placed within Internet Exchange Points and ISP data centers worldwide.
  • This hybrid model optimizes for cost efficiency (especially bandwidth egress), performance, and resilience, allowing Netflix to deliver a superior streaming experience globally.
  • The streaming request flow demonstrates the seamless handoff: AWS orchestrates the session and provides content manifests, but the actual video segments are streamed directly from the nearest OCN appliance.

Understanding this dual infrastructure is fundamental to appreciating the scale and engineering prowess behind Netflix. In the next chapter, we will delve deeper into the specific services and patterns that ensure Netflix’s high availability and fault tolerance, particularly focusing on the role of resilience engineering within its AWS environment.

References

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.