Infrastructure · Note
What actually is an AI cluster?
A cluster is not a pile of accelerators. It is a coordinated machine whose network often decides how useful it is.
An AI cluster is a group of accelerators, servers, network switches, storage systems, and cooling equipment designed to behave like one machine.
The difficult part is coordination. Training distributes a model and its data across many accelerators. Those accelerators repeatedly exchange partial results, often after every layer or batch. If communication is slow, expensive chips sit idle.
That is why cluster design pays so much attention to interconnects: NVLink inside a server, InfiniBand or high-speed Ethernet between servers, and network topology across the facility.
A useful cluster description should answer four questions:
- How many accelerators are available?
- How quickly can they communicate?
- How are jobs scheduled when multiple teams want them?
- What happens when a node fails during a long run?
A rack of GPUs is inventory. A cluster is an operating system, network, and reliability problem wrapped around that inventory.