This review is built from AMD’s published specifications, community benchmarks and third-party testing rather than hands-on use. Figures are attributed in the text; many community numbers are self-reported, and are labelled as such.
EPYC CPU inference means running large language models on AMD EPYC server processors and system memory alone, with no graphics card. It sounds backwards, but it is the cheapest way to hold a model with hundreds of billions of parameters in memory. A 12-channel DDR5 EPYC server can address hundreds of gigabytes of RAM with around 460 to 614 GB/s of bandwidth, which is enough to run giant mixture-of-experts models at usable speeds.
The honest verdict: EPYC CPU inference is the right tool for one job, running very large MoE models such as DeepSeek-class or GLM-class models locally, where no consumer GPU can hold the weights. Expect around 5 to 12 tokens per second on those models, and much faster on MoE models with few active parameters. For anything that fits on a GPU, a GPU is far faster and cheaper per token. And in 2026, RunAIHome warned that the DRAM crisis made the memory cost more than the processors.
Table of Contents
- Why EPYC CPU Inference Works
- Memory Bandwidth Drives EPYC CPU Inference
- EPYC CPU Inference Benchmarks
- MoE Models and EPYC CPU Inference
- Best Generation for EPYC CPU Inference
- Used DDR4 Servers for EPYC CPU Inference
- The RAM Price Problem for EPYC CPU Inference
- Hybrid GPU Plus EPYC CPU Inference
- EPYC CPU Inference Software
- Tuning EPYC CPU Inference: Threads and NUMA
- The Virtual Machine Trap
- A Sample Build
- Serving a Team From One Server
- EPYC CPU Inference vs GPUs and Unified Memory
- Power, Noise and Space
- Who Should Buy What
- FAQ
- Setting Up EPYC CPU Inference
- Verdict
- Sources and Further Reading

Why EPYC CPU Inference Works
Text generation is limited by how fast model weights can be read from memory, not by how much arithmetic a chip can do. A llama.cpp discussion put it simply: LLMs are bound by memory bandwidth, not compute, and faster RAM or a GPU gives more tokens per second.
Consumer desktops have two memory channels and roughly 90 to 100 GB/s of bandwidth, according to RunAIHome. Server platforms have many more channels. AMD notes that 12 DDR5 channels support up to 6TB of RAM per socket. That combination of huge capacity and high bandwidth is what makes a CPU server viable for models far too large for any graphics card.
Memory Bandwidth Drives EPYC CPU Inference
BestLLMfor put it bluntly: memory bandwidth is the only metric that matters, and token throughput tracks it almost linearly. A 12-channel DDR5-4800 EPYC 9004 or 9005 platform delivers about 460 GB/s, roughly 2.7 times an 8-channel DDR4-3200 EPYC 7003 system. AMD’s documentation lists 614 GB/s for 12-channel DDR5-6400.
Two practical rules follow. Fill every memory channel, because an empty channel is lost bandwidth. And buy the fastest memory the platform supports. Core count matters much less than most buyers expect; one test of an EPYC 9175F found decode speed saturating at 13 to 16 threads, even though theoretical bandwidth suggested more headroom.
EPYC CPU Inference Benchmarks
BestLLMfor measured Llama 3.3 70B at Q4_K_M, single user, with llama.cpp.
| CPU | Memory | Bandwidth | Speed | Approximate CPU price |
|---|---|---|---|---|
| EPYC 7763 (Milan, 64 cores) | 8 x DDR4-3200 | 205 GB/s | 3.4 tok/s | $2,200 |
| EPYC 9354P (Genoa, 32 cores) | 12 x DDR5-4800 | 460 GB/s | 7.2 tok/s | $3,800 |
| EPYC 9684X (Genoa-X, 96 cores) | 12 x DDR5-4800, large V-Cache | 460 GB/s | 11.4 tok/s | $6,500 |
| EPYC 9555 (Turin, 64 cores) | 12 x DDR5-6000 | 576 GB/s | 9.8 tok/s | $4,900 |
| EPYC 9755 (Turin, 128 cores) | 12 x DDR5-6400 | 614 GB/s | 10.6 tok/s | $9,200 |
A dense 70B model at under 12 tokens per second is usable but slow. The interesting results come with mixture-of-experts models.
MoE Models and EPYC CPU Inference
A mixture-of-experts model has many parameters but activates only a fraction for each token, so the processor reads far less memory per step. RunAIHome reported a 12-channel EPYC 9454P running a Qwen3 30B MoE model with about 3 billion active parameters at 63.1 tokens per second, and Qwen3-Next 80B at 11.8.
The giant models are where CPUs have no rival. OpenClaw collected owner reports: an EPYC 9654 with 12-channel DDR5 running a 671B model at Q8 at about 6.2 tokens per second, and an older EPYC 7552 with 512GB of DDR4 running a 309GB GLM model at about 5.2. AMD’s own documentation cited 10 to 12 tokens per second for an EPYC 9654 on a sparse MoE model at BF16.
Five to twelve tokens per second on a 600B-class model is slow for chat, but it is local, private, and runs the full model at high precision, which no consumer GPU setup can do.
Best Generation for EPYC CPU Inference
- EPYC 7003 (Milan) with DDR4. Cheapest, eight channels, about 205 GB/s. Good for experimenting with big MoE models slowly.
- EPYC 9004 (Genoa) with DDR5. Twelve channels, about 460 GB/s. The value sweet spot for new builds in many cases.
- EPYC 9004X (Genoa-X). Large V-Cache helped noticeably in BestLLMfor’s 70B test.
- EPYC 9005 (Turin) with DDR5-6000 or 6400. Up to 614 GB/s and up to 192 cores. Fastest, most expensive.
Single-socket systems are simpler and avoid cross-socket memory penalties. Dual-socket systems double capacity and bandwidth on paper, but software must handle NUMA carefully to benefit.
Used DDR4 Servers for EPYC CPU Inference
OpenClaw made the case for used servers: an 8-channel EPYC server with 512GB of registered DDR4 can run a 300 to 460GB mixture-of-experts model that no consumer GPU can touch, at a reported 4 to 6 tokens per second. Ex-datacentre servers and DDR4 modules sell for far less than new DDR5 platforms.
The trade-offs are noise, power and age. Rack servers are loud and draw a lot of power, and older platforms lack the bandwidth of DDR5. For a home lab experimenting with the largest open models, though, a used EPYC server is often the cheapest entry point.
The RAM Price Problem for EPYC CPU Inference
RunAIHome’s headline captured the catch: the EPYC tokens-per-dollar dream meets the DRAM crisis. As memory prices rose sharply in 2026, filling twelve channels with large DDR5 modules became the most expensive part of the build, often costing more than the processor itself.
RunAIHome estimated capital cost per token per second at roughly $115 or more for an EPYC build on a 30B MoE model, against about $12 for a GPU that fits the same model. For models that fit on GPUs, CPUs lose badly on value. The case for EPYC rests entirely on models too large for GPUs.
Hybrid GPU Plus EPYC CPU Inference
A popular middle path combines a CPU server with one GPU. The GPU handles the attention layers and prompt processing, while the large expert weights stay in system RAM. Puget Systems explored this with KTransformers, which advertises up to 28 times faster prefill and 3 times faster decode than llama.cpp; in Puget’s DeepSeek tests, prefill reached roughly 125 to 170 tokens per second and decode around 8 to 14.
The gain in prompt processing is the big win, since CPUs are slow at that phase. If you plan a CPU inference server, budget for at least one decent GPU alongside it.
EPYC CPU Inference Software
BestLLMfor found llama.cpp, and particularly the ik_llama.cpp fork, beat ZenDNN-accelerated vLLM for small-batch interactive use, with vLLM winning only above a batch size of eight. For one person or a small team, a llama.cpp-based stack is the default.
KTransformers suits hybrid CPU and GPU setups for large MoE models. Build llama.cpp with AVX-512 support for Zen 4 and Zen 5, since those instructions help matrix operations on modern EPYC chips.
Tuning EPYC CPU Inference: Threads and NUMA
More threads do not always help. Because the work is limited by memory bandwidth, adding threads beyond the point of saturation adds contention without speed. Test a range of thread counts and keep the fastest.
On dual-socket systems and on chips configured with multiple NUMA nodes, memory placement matters. llama.cpp’s NUMA options, such as distributing memory across nodes, can make a large difference. Clear the system’s file cache before benchmarking, as a widely shared CPU inference guide recommends, to get consistent results.
The Virtual Machine Trap
ServeTheHome described running DeepSeek R1 alongside virtualised workloads on an AMD server, and warned that several of its test setups ran at around 0.1 tokens per second, an order of magnitude slower than expected. Virtual machines can hide memory topology and add overhead that destroys bandwidth-bound performance.
If you must run inference in a VM, pass through memory and CPU topology carefully and pin resources. Bare metal is simpler and faster.
A Sample Build
A sensible single-socket build for EPYC CPU inference looks like this. A Genoa or Turin processor with 32 to 64 cores, because more cores add little once memory bandwidth is saturated. A motherboard with all twelve memory channels exposed. Twelve identical DDR5 RDIMMs at the fastest supported speed, sized so total capacity holds your target model plus a healthy margin for context and the operating system. One GPU with 16 to 24GB for prompt processing and attention layers. Fast NVMe storage large enough for several model files, since 600B-class models at 4-bit occupy hundreds of gigabytes.
Capacity planning is simple arithmetic. A 671B model at 4-bit is roughly 380 to 400GB. At 8-bit, it is around 700GB. Add 10 to 20% for context and overhead. That points to 512GB for 4-bit giants and 768GB or more for 8-bit, which is exactly where memory prices dominate the budget.
Before buying, price the memory first, then the rest. In 2026 that order of operations saves surprises.
Serving a Team From One Server
A single large server can serve a small team, but expectations need care. CPU decode speed is shared: two people generating at once each see roughly half the speed. For a handful of users with occasional requests, that is fine. For a busy team, queues build quickly.
Put a web interface such as Open WebUI or LibreChat in front of the server, so people share one endpoint with proper accounts. Offer a smaller, faster model on a GPU for everyday questions, and reserve the giant model for tasks that need it. That mix makes the expensive server feel responsive for everyone.
EPYC CPU Inference vs GPUs and Unified Memory
| Approach | Model memory | Bandwidth | Best for |
|---|---|---|---|
| EPYC server, 12-channel DDR5 | Hundreds of GB to TBs | 460 to 614 GB/s | 600B-class MoE models |
| Single 32GB GPU | 32GB | Up to 1,792 GB/s | Fast models up to about 32B |
| Mac Studio | Up to several hundred GB | Very high | Large models, quiet and simple |
| DGX Station | 748GB | HBM plus LPDDR5X | Frontier models, at a very high price |
For most people, a GPU or a unified-memory machine is the better choice. EPYC wins only when the model is too large for those options and budget rules out the very top end.
Power, Noise and Space
Server hardware is loud. Rack servers designed for data centres use small, fast fans that are unpleasant in a home or office. Tower workstations built around EPYC are quieter but cost more. Plan for a separate room, or a workstation chassis with large fans.
Power draw under full load is several hundred watts, plus the GPU if you add one. Leaseweb’s tests on an EPYC 9334 showed CPU inference is workable for 7B to 20B models, but an NVIDIA L4 GPU was roughly twice as fast at FP16, which is a reminder that small models belong on GPUs.
Remember too that a CPU server is a general-purpose machine. When it is not running a giant model, it can host virtual machines, databases, build jobs and backups. That flexibility softens the cost in a way a dedicated AI accelerator cannot, and it is a fair reason to choose this route for a home lab or small office server that has several jobs to do.
Who Should Buy What
| If you are | Start with | Why | Link |
|---|---|---|---|
| Wanting to run 600B-class MoE models locally | 12-channel DDR5 EPYC server plus one GPU | Only affordable way to hold the weights | Check price |
| Experimenting cheaply with huge models | Used 8-channel DDR4 EPYC server | Lowest cost per gigabyte | Check price |
| Running models up to about 32B | A GPU | Far faster and cheaper per token | Check price |
| Wanting large models quietly | Mac Studio | High bandwidth, silent | Check price |
Keep firmware and BIOS updated too, since memory compatibility and performance tuning often improve in later releases for server platforms.
Frequently Asked Questions About EPYC CPU Inference
Can a CPU really run a 600B model?
Yes. Owners report around 5 to 6 tokens per second on 600B-class MoE models with large DDR4 or DDR5 memory.
What matters most in the build?
Memory bandwidth: fill every channel with the fastest supported memory.
How fast is a 70B model?
About 3.4 tokens per second on 8-channel DDR4 and 7 to 11 on 12-channel DDR5, in BestLLMfor’s tests.
Is it cheaper than GPUs?
Only for models too large for GPUs. For smaller models, GPUs are far cheaper per token.
Should I add a GPU?
Yes, if possible. Hybrid setups speed up prompt processing dramatically.
Which software should I use?
llama.cpp or the ik_llama.cpp fork for single users, and KTransformers for hybrid CPU and GPU setups.
Setting Up EPYC CPU Inference
Install a recent Linux distribution on bare metal. Populate every memory channel with identical modules, enable the fastest supported memory speed in the BIOS, and check NUMA settings.
Build llama.cpp or ik_llama.cpp with native optimisations, download a quantised MoE model, and test different thread counts. Add a GPU and try KTransformers if prompt processing is too slow.
Finally, measure before you spend. Rent a high-memory cloud server with a similar EPYC processor for a few hours, run your target model with the same software, and record real speeds. That small experiment costs little and tells you whether the speed is acceptable before you commit thousands to hardware and memory.
Verdict
Choose EPYC CPU inference if you want to run the largest open mixture-of-experts models locally and no GPU setup within your budget can hold them. A 12-channel DDR5 server, ideally with one GPU, runs 600B-class models at usable speeds, privately and at high precision.
The honest caveat is speed and cost. Expect single-digit to low double-digit tokens per second on giant models, loud hardware, and in 2026 a memory bill that dominates the budget. For anything that fits on a GPU, buy a GPU.
Next step: decide which giant model you actually need. If it is above 200GB at your chosen quantisation, price a used 8-channel or new 12-channel EPYC build with memory included. If it is smaller, a GPU or Mac Studio is the better path.
Sources and Further Reading
- BestLLMfor: CPU-only inference on EPYC, for measured 70B speeds by generation.
- RunAIHome: CPU vs GPU in 2026, for MoE speeds and the DRAM crisis.
- OpenClaw: used EPYC servers for MoE, for owner-reported giant model speeds.
- Puget Systems: hybrid CPU and GPU inference, for KTransformers results.
- ServeTheHome: DeepSeek R1 on AMD servers, for virtualisation pitfalls.
On this site: plan a server with the home AI server build, compare memory needs in how much VRAM for AI, and see high-end alternatives in our Mac Studio and DGX Station reviews.




