Distributed systems are when computation happens across multiple machines. A single model on a laptop is simple but limited. A model distributed across 100 GPUs in 5 data centers is complex but powerful.
Reasons to distribute: scale (more machines = more capacity), reliability (if one machine fails, others keep working), latency (having servers in multiple locations means users are geographically close), data residency (data stays in certain regions for compliance).
Challenges of distributed systems: coordination (machines need to coordinate and stay consistent), latency (communication between machines is slow compared to memory access), partial failures (network might split, some messages get lost), consistency (if multiple machines update shared state, how do you ensure consistency?).
Models can be distributed in different ways. Data parallelism: the same model runs on multiple machines, each processes different data. This is straightforward and common. Model parallelism: a single model is split across machines (parts of the computation on different machines). This is harder but necessary for very large models.
Inference distribution: requests are distributed across multiple inference servers. Load balancing ensures requests are spread fairly. This enables high throughput.
Training distribution: training data is spread across machines, training happens in parallel. Coordinating training across machines is complex but necessary for large models on large datasets.
Fault tolerance: if a machine fails, the system should recover. This might mean: data is replicated (if one copy is lost, others exist), checkpoints are saved (training can resume from checkpoints), redundancy is built in (more machines than needed, so losing one doesn't hurt).
Consensus protocols solve the problem of keeping distributed machines in sync. Raft and Paxos are popular consensus algorithms. They ensure that when distributed machines need to agree on something (e.g., which model version to use), they can reach agreement even if some machines fail or have slow communication.
Orchestration in distributed systems is complex. You might have Kubernetes managing containers, distributed job frameworks managing training jobs, monitoring systems tracking all machines, logging systems collecting logs from thousands of machines.
Observability is even more critical. When something goes wrong in a distributed system (latency increased, errors appeared), determining the root cause across thousands of machines is hard. Distributed tracing (following requests across machines) is essential.
Cost increases with distribution. Running 100 machines costs more than running 1 machine. You need to ensure the distribution is actually necessary and beneficial.
Most large AI systems are distributed. A single GPU can't handle production workloads. Distribution is non-optional for scale.
Why It Matters
Distributed systems enable scale that single machines can't achieve. They also enable reliability (if one component fails, others compensate). Without distribution, you're limited to single-machine performance.
Example
A recommendation system at scale: model runs on 50 GPU instances across 2 regions, load balancer distributes requests, each instance processes 100 requests/second, data is replicated across regions for reliability, if one region goes down, all traffic goes to the other. Distributed infrastructure makes this possible.