Most AI infrastructure conversations start and end with GPUs. That focus is understandable — accelerators move the bulk of model math. But every inference request also passes through a CPU: it feeds the accelerator, pre-processes the input, runs the routing logic, and serves small models entirely on its own. The CPU for AI inference is one of the most under-scoped decisions in AI procurement, and it is the easiest place to overspend or under-budget.
Based on our market observation across enterprise inference deployments, CPU choice quietly determines three things: how much accelerator idle time you tolerate, how many small-model requests you serve without a GPU at all, and how future-proof your memory and PCIe topology are. This guide covers how to select and source a CPU for AI inference workloads in 2026.
Why the CPU Matters in AI Inference
Inference is fundamentally different from training. Training is a sustained, parallel compute exercise. Inference is latency-sensitive, request-driven, and highly variable. GPUs are superb at the dense matrix math of a single large model, but a production inference service spends a surprising fraction of its time on work that never touches the accelerator:
- Request pre- and post-processing. Tokenization, embedding lookups, output parsing, and API routing all run on the CPU.
- Data feeding. A GPU that starves for data idles at a fraction of its potential. Memory bandwidth and PCIe throughput on the CPU side determine how fast the accelerator gets fed.
- Small-model serving. Many production inference calls are small models — classification, extraction, reranking — where GPU latency and cost are unjustified and the CPU alone is faster end to end.
- Batch orchestration. Schedulers, KV-cache management, and multi-tenant isolation run on the host processor.
What to Look For in an Inference CPU
Not every server CPU is equally good at inference-adjacent work. Prioritize these five characteristics:
- Memory bandwidth. Inference is memory-latency and bandwidth-bound on the CPU path. Eight-channel DDR4 or twelve-channel DDR5 matters far more than raw core count for many serving workloads.
- High single-thread and boost clocks. Pre-processing and small-model latency reward fast cores over sheer core count.
- AI instruction support. Intel AMX (Advanced Matrix Extensions) on 4th/5th Gen Xeon and AMD AVX-512/BF16 support on EPYC give the CPU a genuine on-chip inference capability for INT8/BF16 workloads.
- PCIe lanes. Enough Gen4/Gen5 lanes to attach accelerators, NVMe storage, and high-throughput NICs without contention.
- Core count for consolidation. If the node also runs colocated services, high core density (64–96 cores) lets one host carry inference plus the surrounding stack.
EPYC vs Xeon as an AI Inference CPU
The two x86 platforms approach inference support differently, and both are legitimate — the right choice depends on your workload mix and installed platform base.
AMD EPYC (Genoa and Milan) emphasizes memory bandwidth and core density. Genoa's 12-channel DDR5 delivers roughly 50% more memory bandwidth than a comparable dual-channel platform, which directly reduces accelerator starvation and speeds CPU-only small-model serving. High-core SKUs such as the EPYC 7763 (64 cores, Milan) or Genoa 9004 parts consolidate inference plus the surrounding stack into one host. AVX-512 with BF16 support on EPYC covers INT8/BF16 on-CPU inference cleanly.
Intel Xeon (4th/5th Gen) brings AMX to the table. AMX is a dedicated on-die matrix unit that accelerates INT8/BF16 GEMM operations — the core of transformer inference — without a discrete accelerator. A 32-core 5th Gen Xeon running AMX can serve substantial INT8 workloads on CPU alone, which is why CPU-only inference economics have shifted for small and mid-size models.
In practice, mixed fleets are common: AMD EPYC for high-density, high-bandwidth inference nodes, and Xeon where AMX and existing Intel tooling fit better. Our AMD EPYC and Intel Xeon lines both cover tested refurbished parts.
CPU-Only Inference: When It Makes Sense
The most cost-efficient AI workload is the one that never buys a GPU. CPU-only inference is increasingly practical in four situations:
- Small and mid-size models — classification, reranking, extraction, and encoder models that run in milliseconds on a modern server CPU with AMX or AVX-512/BF16.
- High-throughput, latency-tolerant workloads — batch scoring where aggregate throughput beats per-request latency.
- Edge and branch locations — where GPU power, cooling, and cost are prohibitive and model size is modest.
- GPU overflow and autoscaling — absorbing spikes without provisioning more accelerators.
If your inference mix is 70% small models, a correctly spec'd CPU for AI inference can serve that majority without any GPU spend — reserving accelerators for the large-model minority. This is the single largest budget lever in modern inference architecture.
Building a Balanced Inference Node
When you do pair CPUs with GPUs, the ratio and topology matter. Our GPU server experience points to a few rules of thumb:
- Match CPU bandwidth to accelerator count. A single-socket, 8-channel platform is enough for one or two mid-range GPUs; high-density nodes (4–8 GPUs) want a 12-channel or dual-socket CPU with maximum memory bandwidth.
- Reserve PCIe lanes for accelerators and storage. Confirm the CPU provides Gen5 lanes for GPU and NVMe without sharing.
- Size memory for KV-cache and serving. Inference serving is memory-heavy; buy DDR4/DDR5 capacity generously relative to GPU memory.
- Leave headroom. A 15–20% CPU utilization buffer absorbs request spikes and orchestration overhead without GPU starvation.
Sizing Examples: Two Realistic Inference Nodes
To make the guidance concrete, here are two node archetypes we see repeatedly in production inference fleets. Sizing will vary by model and traffic, but the shape of the build is consistent.
Archetype A — CPU-only INT8 serving node. For classification, reranking, and small encoder workloads at moderate concurrency, a single-socket node with a 32-core 5th Gen Xeon (using AMX) or a 64-core Milan EPYC 7763-class part, 128GB of memory, and two NVMe drives serves tens of thousands of requests per minute without any accelerator. Total platform cost is a fraction of a GPU-equipped rack, and power draw stays under 600W at full load.
Archetype B — GPU-backed generation node. For LLM generation with two to four accelerators, use a dual-socket or 12-channel single-socket CPU with maximum memory bandwidth, 256GB+ system memory for KV-cache and serving overhead, PCIe Gen5 lanes dedicated to the GPUs, and a second NVMe tier for model weights. The CPU here exists to keep accelerators fed: bandwidth and PCIe topology outrank core count.
Both archetypes share a design principle: size the platform for the data path, not just the compute. Inference serving is a pipeline — input parsing, accelerator math, output streaming, and storage all touch the CPU. When the pipeline is balanced, utilization climbs and per-request cost falls; when it is not, the most expensive component in the rack idles.
If you are not sure which archetype fits your workload, the practical route is to profile it: measure concurrent requests, model size, and latency targets, then map those to CPU, memory, and accelerator requirements. Our team regularly walks buyers through this exercise and can suggest a tested refurbished configuration that matches the profile.
Measure before you buy. The cheapest way to validate a CPU choice is telemetry: run representative requests, capture CPU utilization, memory bandwidth saturation, and PCIe throughput, and compare the profile against your latency target. In our experience supporting enterprise buyers, the majority of inference performance surprises trace back to memory bandwidth or PCIe contention, not raw core count — which is why bandwidth-first sizing consistently outperforms core-first sizing.
Keep firmware current. AMX and AVX-512 performance depend on microcode and BIOS revisions. A refurbished node should be updated to the latest supported firmware before deployment; an outdated microcode can silently disable the very instruction sets that justify the CPU choice. We update firmware as part of our testing process and record the revision in the unit's diagnostic report.
Plan the exit. Inference hardware depreciates predictably, and the same three-to-five-year refresh cycle that made today's used EPYC and Xeon parts affordable will apply to the node you build now. Choosing a platform with a deep secondary market — high-volume SKUs on widely deployed sockets — preserves resale value and future upgrade options. This is one more reason mainstream EPYC and Xeon parts outperform exotic or niche SKUs over a full ownership cycle.
Buying Refurbished Inference CPUs
Inference economics reward refurbished hardware more than training does: the workloads are smaller, the latency targets are met easily by previous-generation silicon, and the memory/PCIe platform requirements are already mature. Sourcing decommissioned EPYC and Xeon parts responsibly requires the same discipline as any server CPU buy — OPN verification, matched steppings for 2P, full-core stress testing, and thermal logging. A used 64-core Milan or 32-core 5th Gen Xeon with the right memory config routinely handles production inference mixes at a fraction of new-rack cost.
HKCHL supplies tested refurbished EPYC and Xeon processors for inference builds, confirms platform compatibility, and can match memory and storage to complete the node. If you are sizing an inference deployment and want a practical second opinion on CPU selection or sourcing, our team is happy to discuss your requirements and map a budget-appropriate configuration — whether that means a CPU-only serving node or a balanced GPU-backed build. Contact us for a consultation and quotation.