How to assemble an AI server?

I'll tell a story about how I assembled an AI server for the company's needs and what the outcome was. The company allocated a budget of 125 thousand dollars and said that an AI server must be assembled with this money, which will hold large models for use, as well as for training their models on their own datasets. Prices, of course, are very high, and such a budget is actually not very large to assemble something really serious. Here’s what I collected, and now I will explain my choice and why this configuration was chosen.

Server Components:

• Platform: SuperMicro AS-4125GS-TNRT (SP5, 24 DIMM, 4U)
• Processors: 2 x EPYC 9454 (48 cores / 96 threads)
• RAM: 24 x Kingston Server Premier 32 GB
• GPU Accelerators: 4 x NVIDIA Blackwell 96GB Server Edition
• Hard Drives: 8 x Seagate Exos 10E2400 2.4TB SAS 10K
• HBA Adapter: Broadcom SAS 9400-16e SGL (05-50013-00) PCIe 3.1 x8 LP, Tri-Mode SAS/SATA/NVMe 12G HBA, 16 port (4× ext SFF8644), 3416 IOC
• SSD: 2 × 1600 GB Intel DC P4610 Series Server

Platform

Let's start with the platform. The SuperMicro was chosen because it can accommodate 4–8 GPU accelerators on PCIe 5.0, which is very important. If 4 accelerators are installed, they will fit freely; if 8 are installed, they will be tight, but using NVLink makes it acceptable even in a dense configuration.

Processors

The processors are quite powerful but not excessive in the number of cores since the whole system is assembled for AI, where the CPU plays a more auxiliary role. AMD EPYC are quite good processors with a large number of cores and threads. Each processor has 12 memory channels, totaling 24.

RAM

The RAM is chosen in 32 GB sticks since the SuperMicro platform has 24 slots, and it is preferable to fill them completely. If some slots are left empty and, for example, 64 GB sticks are taken but only 16 installed, it can lead to a loss of about 30–50% of memory bandwidth. The rule is simple — the number of memory channels on the processors should equal the number of memory sticks present.

In the end, we get not one terabyte but 768 GB, trying to stay within budget. This amount is quite sufficient since models mainly operate in VRAM, and only in exceptional cases does offload to RAM occur.

GPU Accelerators

One of the most important components. The choice fell on the NVIDIA Blackwell 96GB Server Edition — rather interesting cards with a large VRAM volume and relatively more affordable compared to A100 or H100. They can hold very large models in memory, and a combination of 4–5 cards can process models up to ~700B parameters.

The difference from more "top-end" solutions is that they lack NVLink, meaning the cards cannot be combined for ultra-fast memory communication. But here PCIe 5.0 comes to the rescue — all cards are installed on it and exchange data at high speed, partially compensating for the absence of NVLink.

Hard Drives and HBA

Server SAS hard drives of 10 TB are installed — a total of 8, combined in RAID 5. This results in about 30 TB of usable space, used for datasets. Since there are various models, and each can weigh hundreds of gigabytes, this volume is quite enough.

Also, two server SSDs in RAID 1 are installed for the system. SSDs were chosen over NVMe for a simple reason: server SSDs are more durable (up to ~3.5 PB of writes), while NVMe are more expensive, and their speed is not critical in this scenario.

Overall, this is the AI server that was built. I don't know, maybe it will be useful to someone if they are gathering a similar machine. If there are any questions, feel free to ask in the comments, and I will respond.