Foundations of
System Design

Foundation 01

Client-Server Architecture

Before anything else, understand the fundamental model that powers virtually every application you've ever used. A client is any process that initiates a request. A server is any process that responds to it. The key insight is that these are roles, not fixed machines.

The Request-Response Cycle

Every interaction follows a pattern: client sends a request, server processes it, server sends a response. Understanding this cycle deeply — including what happens in each step, what can fail, and what adds latency — is the foundation for diagnosing every system problem you'll encounter.

Client                Network              Server
  |                     |                    |
  |--- HTTP GET /user -->|                    |
  |                     |--- TCP segment --->|
  |                     |                    |-- Parse request
  |                     |                    |-- Auth check
  |                     |                    |-- DB query
  |                     |<-- TCP segment ----|
  |<-- HTTP 200 JSON ---|                    |
  |                     |                    |

What Can Go Wrong

Foundation 02

Monolith vs Microservices

This is one of the most misunderstood topics in the industry. Teams rush to microservices without understanding the costs. Let's think through both options rigorously.

When Microservices Make Sense

Microservices solve organisational problems more than technical ones. When you have 200 engineers all deploying to the same codebase, deployments become a coordination nightmare. Microservices let teams own their services independently.

The Real Costs of Microservices

• Network latency: In-process calls take ~100ns. Network calls take ~1ms. That's 10,000x slower — per hop.
• Distributed transactions: You lose ACID guarantees. You now need sagas, two-phase commit, or eventual consistency.
• Operational overhead: Each service needs its own CI/CD, monitoring, logging, scaling policies, health checks.
• Debugging complexity: A bug might span 6 services. Distributed tracing becomes essential.
• Data consistency: Each service owns its data. Cross-service queries become API calls, not SQL joins.

Foundation 03

Horizontal vs Vertical Scaling

The Stateless Requirement

Horizontal scaling requires your servers to be stateless — any server must be able to handle any request. This means session state must live outside the server: in a database, Redis, or a JWT token that the client carries.

Foundation 04

Latency vs Throughput

Two of the most important metrics in system design — and they're often in tension with each other.

Throughput vs Latency Tradeoff

Throughput is how many requests you can handle per second. Latency is how long each request takes. These can conflict:

• Batching increases throughput but increases latency (you wait to fill a batch)
• Caching reduces latency but doesn't always improve throughput
• More connections can increase throughput but also increase contention and latency

Foundation 05

CAP Theorem

The CAP Theorem states that a distributed system can guarantee at most two of these three properties simultaneously: Consistency, Availability, and Partition Tolerance.

Why "CA" Doesn't Exist in Practice

A system that sacrifices partition tolerance assumes the network never fails. Networks always fail. Every distributed system must be partition tolerant. So the real choice is: CP or AP when a partition happens.

PACELC: A Better Model

The PACELC theorem extends CAP: even when the system is running normally (no partition), there's a tradeoff between Latency and Consistency. Strong consistency requires waiting for all nodes to agree, which adds latency. This is the tradeoff that Cassandra, DynamoDB, and others expose through tunable consistency levels.

Foundation 06

ACID vs BASE

Isolation Levels Deep Dive

Most engineers underestimate how much ACID's Isolation property costs. The four isolation levels (weakest to strongest):

• Read Uncommitted: Reads dirty data. Almost never used. Maximum concurrency.
• Read Committed: Only sees committed data. Default in most DBs. Prevents dirty reads but allows non-repeatable reads.
• Repeatable Read: Same query returns same results within a transaction. Prevents phantom reads in MySQL (with gap locks). Default in MySQL/InnoDB.
• Serializable: Full isolation. Transactions behave as if serial. Heaviest locking. Lowest throughput.

Foundation 07

Consistency Models

Consistency is a spectrum. Understanding where each model sits — and what guarantees it provides — is essential for designing distributed systems.

• Linearisability (Strong Consistency): Every read returns the most recently written value. Operations appear instantaneous. Requires coordination (Raft, Paxos). High latency cost. Examples: ZooKeeper, etcd, Google Spanner.
• Sequential Consistency: Operations appear in the same order to all nodes, but not necessarily in real-time order. Weaker than linearisability. Hard to implement efficiently.
• Causal Consistency: Operations that are causally related are seen in the same order by all nodes. Independent operations may be seen in any order. MongoDB sessions.
• Eventual Consistency: Given no new updates, all replicas eventually agree. Reads may return stale data in the interim. DynamoDB, Cassandra, DNS. High availability, low latency.
• Read-Your-Writes: After a write, you always see your own write. Even if other nodes are stale. Session consistency. User profile updates must have this.

Foundation 08

Replication

Replication means keeping copies of data on multiple nodes. It serves two purposes: fault tolerance (if one node dies, data survives) and performance (read from the nearest replica).

Foundation 09

Partitioning & Sharding

When a single database becomes too large or too slow, you split the data across multiple nodes. Each piece is called a shard or partition.

Partitioning Strategies

• Range-based: Users A–M on Shard 1, N–Z on Shard 2. Simple but prone to hot spots if distribution is uneven.
• Hash-based: hash(user_id) % N shards. Even distribution but makes range queries expensive (data is scattered).
• Consistent hashing: Hash both keys and nodes onto a ring. Minimises data movement when nodes are added/removed. Used by: DynamoDB, Cassandra, Chord.
• Directory-based: A lookup table maps keys to shards. Flexible but the directory is a bottleneck and single point of failure.

The Hot Partition Problem

If 10% of your users generate 90% of your traffic (celebrity accounts, viral posts), they land on the same shard. That shard becomes overwhelmed while others are idle. Solutions:

• Add a random suffix to hot keys (user_123_<0-9>) and fan out reads
• Cache hot data in front of the database
• Use a separate "celebrity" tier with different hardware

Foundation 10

Load Balancing

A load balancer distributes incoming traffic across multiple servers. It sits between clients and your server pool, providing: horizontal scaling, fault tolerance, and session management.

Load Balancing Algorithms

• Round Robin: Send request 1 to server A, 2 to B, 3 to C, cycle. Simple, ignores server load. Bad if servers are heterogeneous.
• Weighted Round Robin: More powerful servers get more requests. Better resource utilisation.
• Least Connections: Send to the server with fewest active connections. Better than round robin when requests have variable duration.
• Least Response Time: Combine fewest connections + fastest response time. Most intelligent but requires monitoring overhead.
• IP Hash: Hash client IP to always route to the same server. Useful for session affinity (sticky sessions). Problem: if server dies, those sessions are lost.
• Consistent Hashing: Used by CDNs and distributed caches. Minimises reassignment when servers change.

Layer 4 vs Layer 7 Load Balancing

Foundation 11

Reverse Proxy & API Gateway

Foundation 12

Content Delivery Network (CDN)

A CDN is a globally distributed network of cache servers (edge nodes) that serve content from the location nearest to the user. This reduces latency from ~150ms (cross-continent) to ~10ms (local edge node).

Push vs Pull CDN

Foundation 13

Rate Limiting

Rate limiting controls how many requests a client can make in a time window. Essential for: preventing abuse, preventing DDoS, ensuring fair usage, and protecting downstream services from being overwhelmed.

Algorithms

• Token Bucket: Bucket holds N tokens. Each request consumes one token. Tokens replenish at a constant rate. Allows short bursts (up to bucket size) while maintaining a long-term rate. Used by: most APIs (Twitter, Stripe).
• Leaky Bucket: Requests enter a queue (bucket) and are processed at a fixed rate. Smooths out bursts. If the queue is full, requests are dropped. Good for traffic shaping.
• Fixed Window Counter: Count requests per fixed window (e.g., 100 req/minute). Simple. Vulnerable to boundary attacks (100 at 00:59 + 100 at 01:00 = 200 in 2 seconds).
• Sliding Window Log: Track timestamps of all requests. Check if N requests in the past 60s. Accurate but memory-intensive.
• Sliding Window Counter: Hybrid: use current + previous window weighted by overlap. Good accuracy at low memory cost.

Foundation 14

Circuit Breakers

A circuit breaker is a pattern that prevents cascading failures. When a downstream service starts failing, the circuit breaker "opens" and stops sending requests to it, giving it time to recover.

Three States

• Closed (normal): Requests flow through. Failures are counted. If failures exceed threshold, transition to Open.
• Open (tripped): All requests immediately fail (fast fail) without hitting the downstream service. After a timeout, transition to Half-Open.
• Half-Open (testing): Allow a small number of trial requests. If they succeed, transition to Closed. If they fail, back to Open.

Foundation 15

Idempotency

An operation is idempotent if calling it multiple times produces the same result as calling it once. Critical for distributed systems where retries are inevitable.

Making Operations Idempotent

• HTTP GET/PUT/DELETE are naturally idempotent by definition. POST is not.
• Idempotency keys: Client sends a unique key with each request. Server stores the key + result. If the key is seen again, return the cached result without reprocessing.
• Conditional updates: Use version numbers or ETags. "Update record only if version = 5." If version has changed, reject the update.
• Natural idempotency: "Set balance to £100" is idempotent. "Add £10 to balance" is not. Design APIs to use set semantics where possible.

Foundation 16

Fault Tolerance & Availability

The Nines of Availability

Strategies for High Availability

• Redundancy: Multiple instances of every component. No single point of failure.
• Health checks + auto-restart: Load balancers probe health endpoints and route around unhealthy nodes.
• Graceful degradation: If recommendations service is down, show popular items. Don't make the whole page fail.
• Chaos engineering: Intentionally kill services in production to test failover (Netflix Chaos Monkey).
• Geographic redundancy: Run in multiple regions. If us-east-1 goes down, failover to eu-west-1.