High-Performing and Scalable Network Architectures (Task 3.4)

High-Performing and Scalable Network Architectures

Source: https://docs.aws.amazon.com/wellarchitected/latest/performance-efficiency-pillar/

Network performance is a LATENCY and PATH problem: every hop (public internet, cross-AZ, cross-Region, load balancer tier) adds time and cost. The design discipline is choosing where traffic enters AWS, how it moves inside, and which load balancer layer matches the protocol.

Load Balancer Selection

How to reason it: the balancer layer is chosen by what you must SEE. Need paths, headers, cookies, hostnames (routing, auth, redirects) → ALB at L7. Need raw throughput, static IPs, UDP, or the true client IP (NLB preserves it; ALB uses X-Forwarded-For) → NLB at L4. Need third-party appliances in-path for ALL traffic → GWLB tunneling.

Real use-case: A gaming backend tried an ALB — but console clients needed fixed IPs for firewall allow-listing, the protocol was UDP, and anti-cheat needed the real player IP. The NLB gave static EIPs, native UDP, and preserved source IPs. Same app's admin portal stayed on the ALB for path-based routing and Cognito auth — both balancers, each doing its layer's job.

Gotchas & interview notes: "millions of requests per second" / "static IP" / "preserve source IP" → NLB. "route /api to one service, /admin to another" → ALB. Cross-zone load balancing: ALB ON by default (even spread, small cross-AZ charges); NLB OFF by default (AZ-local, latency-optimized) — know which default serves which goal.

Edge and Global Performance

How the two edge services differ:

• CloudFront: caches at 400+ PoPs — for cacheable HTTP(S); also accelerates dynamic content (keep-alive over the AWS backbone) and APIs
• AWS Global Accelerator: static anycast IPs; users enter the AWS edge closest to them and ride the private backbone to your Regional endpoints — best for non-HTTP protocols (TCP/UDP gaming, IoT) or when clients need fixed IPs; preserves client IP with NLB
• Route 53: latency-based/geoproximity routing; health-check failover; latency measured continuously
• S3 Transfer Acceleration: fast uploads over the edge network
• AWS PrivateLink / VPC interface endpoints: expose/consume services with private IPs — no IGW, no peering, no public internet; used for SaaS-to-VPC and inter-account APIs
• Gateway endpoints (S3, DynamoDB): free, route-table-based private access
• VPC peering = 1:1, non-transitive; Transit Gateway = hub-and-spoke for many VPCs + VPN/DX (transitive, scales, inter-Region peering available)
• Multi-AZ subnets per tier (public web / private app / isolated data), sized CIDRs with growth headroom (/16 VPC → /20 tiers → /24 subnets)
• Route tables per tier (public → IGW; private → NAT; data → local only)
• IP addressing hygiene: avoid overlapping CIDRs with future VPC peering/VPN partners
• Cross-AZ traffic bills both ways; AZ-affinity designs (app + its cache/database in the same AZ) trade resilience for latency/cost when needed
• Data transfer OUT to internet is the priciest path — front everything with CloudFront (edge pricing) when serving users
• Route 53 latency routing → 3 Regional deployments (us/eu/ap)
• Global Accelerator anycast IPs (fixed IPs for console clients, TCP/UDP, non-HTTP) → NLB per Region (static IPs, preserves player IP for anti-cheat, millions of concurrent flows)
• Session servers in private subnets; game state in Region-local MemoryDB
• Corporate HQ office links via Direct Connect 10 Gbps with VPN failover for management plane
• Cross-Region replication of player profiles (Aurora Global)
• Player-facing downloads served by CloudFront

The deciding question: can the traffic be CACHED? If content repeats (static assets, API GETs), CloudFront serves copies from edge PoPs — the closer copy IS the optimization. If traffic is unique per request (game sessions, IoT telemetry, UDP), caching does nothing — Global Accelerator still buys the anycast entry + private backbone path, bypassing congested public hops.

Real use-case: A global SaaS serves both a web console and a TCP-based data-sync agent. Console traffic → CloudFront (cacheable, TLS termination, WAF). Agent traffic → Global Accelerator anycast IPs (fixed IPs for partner allow-lists, UDP, non-HTTP). Users in Sydney reach the us-east-1 backend over the AWS backbone in 100 ms instead of 250 ms of public internet — the two edge services split the traffic by cacheability.

Gotchas & interview notes: "non-HTTP/TCP/UDP latency" → Global Accelerator; "cacheable content for global users" → CloudFront. Global Accelerator also gives instant client-side failover between Regions (health-checked) — its IPs do not change when endpoints do.

Hybrid Connectivity Performance

How to reason it: VPN rides the public internet — cheap, instant, variable (jitter matters for VoIP/replication). Direct Connect is a private circuit — consistent latency, high bandwidth, weeks of carrier lead time, NO built-in redundancy (a fiber cut is an outage). Production hybrid = DX primary + VPN failover, or dual DX at separate DX locations.

Real use-case: Nightly 2 TB replication over VPN kept missing its window (variable throughput). A 1 Gbps Direct Connect moved it to a steady 4-hour transfer; the VPN tunnel stayed as failover and proved itself during a scheduled DX maintenance — the composite pattern is the answer, not either alone.

Gotchas & interview notes: "consistent, predictable latency" → DX; "quick, cost-effective connection" → VPN; "resilient hybrid" → DX + VPN. Direct Connect does not encrypt by default — add MACsec or IPsec over it for regulated traffic.

Private Service Exposure

• CloudFront: caches at 400+ PoPs — for cacheable HTTP(S); also accelerates dynamic content (keep-alive over the AWS backbone) and APIs
• AWS Global Accelerator: static anycast IPs; users enter the AWS edge closest to them and ride the private backbone to your Regional endpoints — best for non-HTTP protocols (TCP/UDP gaming, IoT) or when clients need fixed IPs; preserves client IP with NLB
• Route 53: latency-based/geoproximity routing; health-check failover; latency measured continuously
• S3 Transfer Acceleration: fast uploads over the edge network
• AWS PrivateLink / VPC interface endpoints: expose/consume services with private IPs — no IGW, no peering, no public internet; used for SaaS-to-VPC and inter-account APIs
• Gateway endpoints (S3, DynamoDB): free, route-table-based private access
• VPC peering = 1:1, non-transitive; Transit Gateway = hub-and-spoke for many VPCs + VPN/DX (transitive, scales, inter-Region peering available)
• Multi-AZ subnets per tier (public web / private app / isolated data), sized CIDRs with growth headroom (/16 VPC → /20 tiers → /24 subnets)
• Route tables per tier (public → IGW; private → NAT; data → local only)
• IP addressing hygiene: avoid overlapping CIDRs with future VPC peering/VPN partners
• Cross-AZ traffic bills both ways; AZ-affinity designs (app + its cache/database in the same AZ) trade resilience for latency/cost when needed
• Data transfer OUT to internet is the priciest path — front everything with CloudFront (edge pricing) when serving users
• Route 53 latency routing → 3 Regional deployments (us/eu/ap)
• Global Accelerator anycast IPs (fixed IPs for console clients, TCP/UDP, non-HTTP) → NLB per Region (static IPs, preserves player IP for anti-cheat, millions of concurrent flows)
• Session servers in private subnets; game state in Region-local MemoryDB
• Corporate HQ office links via Direct Connect 10 Gbps with VPN failover for management plane
• Cross-Region replication of player profiles (Aurora Global)
• Player-facing downloads served by CloudFront

How to reason it: the question is WHO needs to reach WHAT privately. One VPC → another VPC: peering (non-transitive — A-B and B-C does NOT give A-C). Many VPCs, hub-and-spoke, transitive routing: Transit Gateway. Consumers outside your network (other accounts, SaaS partners) reaching YOUR service: PrivateLink — traffic never touches the internet and neither side joins the other's network.

Real use-case: 40 production VPCs needed S3 access without internet routing — gateway endpoints (free, route-table entry) on every VPC. Partner SaaS consumed the company's internal API over PrivateLink: private IPs both sides, no VPC peering mesh (40² / 2 peerings avoided), and no data transfer charges beyond the endpoint hours.

Gotchas & interview notes: "traffic to S3 must not traverse the internet" → gateway endpoint (free); "consume a partner's API privately" → PrivateLink. Peering is non-transitive — a Transit Gateway question hides in any 3+ VPC topology.

Network Topology Design

• CloudFront: caches at 400+ PoPs — for cacheable HTTP(S); also accelerates dynamic content (keep-alive over the AWS backbone) and APIs
• AWS Global Accelerator: static anycast IPs; users enter the AWS edge closest to them and ride the private backbone to your Regional endpoints — best for non-HTTP protocols (TCP/UDP gaming, IoT) or when clients need fixed IPs; preserves client IP with NLB
• Route 53: latency-based/geoproximity routing; health-check failover; latency measured continuously
• S3 Transfer Acceleration: fast uploads over the edge network
• AWS PrivateLink / VPC interface endpoints: expose/consume services with private IPs — no IGW, no peering, no public internet; used for SaaS-to-VPC and inter-account APIs
• Gateway endpoints (S3, DynamoDB): free, route-table-based private access
• VPC peering = 1:1, non-transitive; Transit Gateway = hub-and-spoke for many VPCs + VPN/DX (transitive, scales, inter-Region peering available)
• Multi-AZ subnets per tier (public web / private app / isolated data), sized CIDRs with growth headroom (/16 VPC → /20 tiers → /24 subnets)
• Route tables per tier (public → IGW; private → NAT; data → local only)
• IP addressing hygiene: avoid overlapping CIDRs with future VPC peering/VPN partners
• Cross-AZ traffic bills both ways; AZ-affinity designs (app + its cache/database in the same AZ) trade resilience for latency/cost when needed
• Data transfer OUT to internet is the priciest path — front everything with CloudFront (edge pricing) when serving users
• Route 53 latency routing → 3 Regional deployments (us/eu/ap)
• Global Accelerator anycast IPs (fixed IPs for console clients, TCP/UDP, non-HTTP) → NLB per Region (static IPs, preserves player IP for anti-cheat, millions of concurrent flows)
• Session servers in private subnets; game state in Region-local MemoryDB
• Corporate HQ office links via Direct Connect 10 Gbps with VPN failover for management plane
• Cross-Region replication of player profiles (Aurora Global)
• Player-facing downloads served by CloudFront

Placement Economics

• CloudFront: caches at 400+ PoPs — for cacheable HTTP(S); also accelerates dynamic content (keep-alive over the AWS backbone) and APIs
• AWS Global Accelerator: static anycast IPs; users enter the AWS edge closest to them and ride the private backbone to your Regional endpoints — best for non-HTTP protocols (TCP/UDP gaming, IoT) or when clients need fixed IPs; preserves client IP with NLB
• Route 53: latency-based/geoproximity routing; health-check failover; latency measured continuously
• S3 Transfer Acceleration: fast uploads over the edge network
• AWS PrivateLink / VPC interface endpoints: expose/consume services with private IPs — no IGW, no peering, no public internet; used for SaaS-to-VPC and inter-account APIs
• Gateway endpoints (S3, DynamoDB): free, route-table-based private access
• VPC peering = 1:1, non-transitive; Transit Gateway = hub-and-spoke for many VPCs + VPN/DX (transitive, scales, inter-Region peering available)
• Multi-AZ subnets per tier (public web / private app / isolated data), sized CIDRs with growth headroom (/16 VPC → /20 tiers → /24 subnets)
• Route tables per tier (public → IGW; private → NAT; data → local only)
• IP addressing hygiene: avoid overlapping CIDRs with future VPC peering/VPN partners
• Cross-AZ traffic bills both ways; AZ-affinity designs (app + its cache/database in the same AZ) trade resilience for latency/cost when needed
• Data transfer OUT to internet is the priciest path — front everything with CloudFront (edge pricing) when serving users
• Route 53 latency routing → 3 Regional deployments (us/eu/ap)
• Global Accelerator anycast IPs (fixed IPs for console clients, TCP/UDP, non-HTTP) → NLB per Region (static IPs, preserves player IP for anti-cheat, millions of concurrent flows)
• Session servers in private subnets; game state in Region-local MemoryDB
• Corporate HQ office links via Direct Connect 10 Gbps with VPN failover for management plane
• Cross-Region replication of player profiles (Aurora Global)
• Player-facing downloads served by CloudFront

Real use-case: A trading API discovered 40% of its latency budget was cross-AZ hops: app tier in AZ-a, Redis in AZ-b, and the NLB balancing across both. Pinning the request path to AZ-affinity (each AZ's app tier uses its own Redis replica) cut p99 by 8 ms and cross-AZ charges by 90% — an informed trade: resilience now depends on per-AZ replication instead of cross-AZ calls.

Worked Example: Global Game Backend

• CloudFront: caches at 400+ PoPs — for cacheable HTTP(S); also accelerates dynamic content (keep-alive over the AWS backbone) and APIs
• AWS Global Accelerator: static anycast IPs; users enter the AWS edge closest to them and ride the private backbone to your Regional endpoints — best for non-HTTP protocols (TCP/UDP gaming, IoT) or when clients need fixed IPs; preserves client IP with NLB
• Route 53: latency-based/geoproximity routing; health-check failover; latency measured continuously
• S3 Transfer Acceleration: fast uploads over the edge network
• AWS PrivateLink / VPC interface endpoints: expose/consume services with private IPs — no IGW, no peering, no public internet; used for SaaS-to-VPC and inter-account APIs
• Gateway endpoints (S3, DynamoDB): free, route-table-based private access
• VPC peering = 1:1, non-transitive; Transit Gateway = hub-and-spoke for many VPCs + VPN/DX (transitive, scales, inter-Region peering available)
• Multi-AZ subnets per tier (public web / private app / isolated data), sized CIDRs with growth headroom (/16 VPC → /20 tiers → /24 subnets)
• Route tables per tier (public → IGW; private → NAT; data → local only)
• IP addressing hygiene: avoid overlapping CIDRs with future VPC peering/VPN partners
• Cross-AZ traffic bills both ways; AZ-affinity designs (app + its cache/database in the same AZ) trade resilience for latency/cost when needed
• Data transfer OUT to internet is the priciest path — front everything with CloudFront (edge pricing) when serving users
• Route 53 latency routing → 3 Regional deployments (us/eu/ap)
• Global Accelerator anycast IPs (fixed IPs for console clients, TCP/UDP, non-HTTP) → NLB per Region (static IPs, preserves player IP for anti-cheat, millions of concurrent flows)
• Session servers in private subnets; game state in Region-local MemoryDB
• Corporate HQ office links via Direct Connect 10 Gbps with VPN failover for management plane
• Cross-Region replication of player profiles (Aurora Global)
• Player-facing downloads served by CloudFront

The senior summary: choose the balancer by what you must inspect, the edge service by cacheability, hybrid links by consistency-vs-speed, and private connectivity by who-consumes-what — then count cross-AZ hops like money, because they are.