Work · Inference economics

    What should run your models?

    Start with the model and the service you need to deliver. GPUs offer flexibility; Mac Studio makes local inference accessible; Korean NPUs deserve a workload-specific trial. The decision is accepted tokens per dollar at a usable response time.

    Reviewed · 8B · 70B · 120B+ / MoE · USD prices

    First: will the model fit?

    These are arithmetic lower bounds for weights: parameters × bits ÷ 8, in decimal GB. Add quantization metadata, KV cache, runtime buffers and OS headroom. Longer context and more concurrent requests consume more memory. A mixture-of-experts model still needs storage for all its experts, even when fewer are active per token.

    Weight-only memory estimates, not serving requirements
    Model sizeBF16 / FP168-bit4-bit
    8B16 GB8 GB4 GB
    70B140 GB70 GB35 GB
    120B240 GB120 GB60 GB
    671B1,342 GB671 GB335.5 GB

    8B

    Start with a single accelerator or local Mac. Eight-GPU results below represent aggregate serving capacity, not a minimum purchase.

    70B

    Quantization can make a large difference to fit. Compare quality after quantization; 35 GB of 4-bit weights does not mean a 35 GB device can serve the model.

    120B+ / large MoE

    Check sharding, interconnect and supported operators. Aggregate card memory is not one automatically shared pool; fitting the weights does not prove useful throughput.

    Five vendors, different buying decisions

    Purchase prices are observed US offers, not Korea landed costs or a market average. Tax, shipping, exchange rates, network switches, racks and facility work need a local quote. Ratings below are specifications, not a shared power benchmark.

    NVIDIA GPU · B200 / B300

    Eight-GPU rack server; the B200 price and DGX specifications below describe different systems.

    Accelerator memory
    DGX B200: 1,440 GB GPU memory. Selected B300 benchmark nodes: 8 × 270 GB usable.
    Purchase cost
    $374,687.50: Exeton listed B200 configuration, 2 × EPYC 9454, 768 GB host RAM, 3.84 TB SSD, 3-year warranty. B300: obtain a configuration-specific quote.
    Power & cooling
    DGX B200: ~14.3 kW maximum system rating. This is not measured benchmark power or the Exeton server's specification.
    Where to evaluate it
    Broad model experimentation and high-concurrency serving. CUDA / TensorRT-LLM configurations have published benchmark evidence; validate the exact runtime and model before buying.
    NVIDIA DGX B200 specifications ↗Exeton B200 server configuration ↗MLCommons Datacenter results & methodology ↗

    AMD GPU · Instinct MI355X

    Eight-GPU server with direct liquid cooling; ROCm / vLLM serving. MI400 and Helios have since launched. This comparison retains the measured MI355X configuration, not an estimate of MI400 performance.

    Accelerator memory
    288 GB HBM3e per GPU; 2,304 GB across eight. Memory bandwidth: up to 8 TB/s per GPU.
    Purchase cost
    Server purchase: quote required; no verified public full-system price in this review. A spot rental reference appears below.
    Power & cooling
    1.4 kW per GPU: 11.2 kW for eight accelerators alone. Add host and cooling; this is not wall power.
    Where to evaluate it
    A serious GPU alternative for supported models. Measure kernel coverage, quantization, distributed serving and operating effort on your own ROCm stack.
    AMD MI350 series architecture ↗AMD MI400 / Helios launch disclosure · August 4, 2026 ↗DigitalOcean GPU pricing ↗MLCommons Datacenter results & methodology ↗

    Apple Mac Studio · M5 Max / M5 Ultra

    Desktop for local inference; MLX uses shared CPU/GPU memory. Neural Engine TOPS is not an LLM serving benchmark.

    Accelerator memory
    M5 Max: up to 128 GB. Apple's specifications checked October 27 list M5 Ultra with 96 GB, configurable to 256 / 512 GB with the 80-core GPU. Ultra bandwidth: 1.2 TB/s. OS and runtime share this memory; check regional order availability.
    Purchase cost
    US starting prices: M5 Max $2,499; M5 Ultra $5,499. Those are base configurations, not 256 / 512 GB prices. Confirm current memory-option availability.
    Power & cooling
    No comparable wall-power / tokens-per-second pair verified for these M5 configurations. Measure at the outlet.
    Where to evaluate it
    Local development and modest concurrency, especially quantized models that fit unified memory. Capacity alone does not establish latency or datacenter reliability.
    Apple Mac Studio announcement · 2026-08-25 ↗Apple Mac Studio specifications ↗Apple MLX · unified memory ↗

    Rebellions (리벨리온) · ATOM-Max / RebelServer

    ATOM-Max: 4U, up to eight cards. Newer RebelServer: 5U, eight RebelCards with Rebel100 chips.

    Accelerator memory
    ATOM-Max: 512 GB GDDR6 across eight cards. RebelServer's listed 1.5 TB is host DDR5; do not treat it as accelerator memory.
    Purchase cost
    Both: sales quote required. Request hardware, SDK/support, model porting and installation as separate items.
    Power & cooling
    Vendor ratings: ATOM-Max ~3.4 kW typical / 4.3 kW max; RebelServer typical 4–6 kW / 7 kW max. Not matched workload measurements.
    Where to evaluate it
    Pilot the exact model with the RBLN serving stack. SKT deployment is evidence of commercial use; a 500B A.X K1 demo on RebelServer establishes model execution, not a throughput guarantee.
    Rebellions ATOM-Max Server ↗Rebellions RebelServer ↗RebelServer A.X K1 demonstration · 2026-07-23 ↗Rebellions / SKT deployment · 2026-09-21 ↗

    FuriosaAI (퓨리오사AI) · NXT RNGD Server

    Air-cooled 4U, eight RNGD accelerators, Furiosa SDK.

    Accelerator memory
    384 GB HBM3 across eight cards; 1 TB host DDR5 is separate.
    Purchase cost
    Sales quote required. No verified public server price in this review.
    Power & cooling
    3 kW published system envelope; benchmark efficiency below uses scaled server envelopes.
    Where to evaluate it
    Evaluate supported production models. The September Upstage case describes Solar Pro 3 API / Daum search serving alongside H200, with two four-card replicas per server.
    Furiosa NXT RNGD specifications & benchmark ↗Furiosa / Upstage production case · September 2026 ↗

    Throughput, with the test attached

    Selected MLPerf Inference v6.1 Closed / Available / Server submissions: one node, eight accelerators each. These are aggregate output tokens/second under the benchmark's model-specific quality and latency rules, not single-user speed. Compare within one model only; the software and precision also differ.

    Llama 2 70B · 99.9% quality · Server · 8 accelerators / node
    Submitted systemOutput tok/sWeightsSource
    Cisco · NVIDIA B300-SXM-270GBCisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT)Not submitted—6.1-0014 ↗
    CoreWeave · NVIDIA B200-SXM-180GBCoreWeave B200 SXM (8x B200-SXM-180GB)102,398FP46.1-0021 ↗Configuration
    Supermicro · AMD Instinct MI355X 288GB HBM3eAS-4126GS-NMR-LCC103,032FP46.1-0099 ↗Configuration
    Supermicro · NVIDIA B300-SXM-270GBAS-8126GS-NB3RT106,872FP46.1-0100 ↗Configuration

    The Supermicro pair keeps submitter and accelerator count constant. Cisco adds an 8B B300 result; CoreWeave supplies a B200 reference. Missing cells remain missing. These systems are not the quoted retail configuration. No cross-vendor performance-per-watt claim is derived from their TDPs.

    Software & pinned source snapshot
    • 6.1-0014: TensorRT 10.14, CUDA 13.1
    • 6.1-0021: TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0
    • 6.1-0099: vLLM 0.22.0, PyTorch 2.10.0, ROCm 7.2.4
    • 6.1-0100: TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentML:mlperf-inf-mm-q3vl-v6.0
    MLCommons summary.xlsx · 4bb63cd28eb1 ↗
    MLCommons Datacenter results & methodology ↗MLPerf model, quality & latency rules ↗

    Mac Studio: measure the serving workload

    No comparable 8B / 70B / large-model throughput verified for the M5 systems here. Apple's relative AI-compute claim cannot be converted into tokens/second. Record prefill speed and decode speed separately with MLX or your chosen runtime.

    Rebellions: absolute throughput still needed

    The ATOM-Max brief compares Llama 3 70B with an unnamed GPU using relative throughput. It does not establish a comparable absolute result. SKT's 14M requests/day is fleet traffic, not one server's capacity; the RebelServer 500B demo also lacks an absolute serving result.

    ATOM-Max product brief · July 2025 ↗Rebellions / SKT deployment · 2026-09-21 ↗RebelServer A.X K1 demonstration · 2026-07-23 ↗

    Furiosa: a different, vendor-run test

    Qwen3-32B FP8, 1K input / 1K output: 779 tok/s/kW at 37 tok/s/user. SDK 2026.4 versus RTX PRO 6000 with vLLM 0.28.0; four-card measurements scaled to eight-card power envelopes (3 / 6.5 kW). This is not an MLPerf 70B result or measured eight-card wall efficiency.

    Furiosa NXT RNGD specifications & benchmark ↗

    Rent before you commit

    DigitalOcean public prices checked on the review date. Eight-GPU costs below are arithmetic multiples, not a separate quote. Spot capacity can be reclaimed; verify availability and minimum topology. A powered-off reserved instance can still incur charges until destroyed.

    AcceleratorTerms1 GPU / hour8 GPUs / hour8 GPUs × 730 hours
    AMD MI300XOn-demand$2.59$20.72$15,125.60
    NVIDIA H200On-demand$4.47$35.76$26,104.80
    AMD MI355XInterruptible spot$5.50$44.00$32,120.00
    NVIDIA B300Interruptible spot$6.17$49.36$36,032.80
    DigitalOcean GPU pricing ↗

    Compare the cost of useful output

    The starting numbers are hypothetical examples, not vendor quotes or measured GPU/NPU results. Replace both systems with quotes and goodput measured on the same model, quality, prompt lengths and latency target. Inputs stay in this browser session.

    System A

    Cost / million accepted output tokens

    $1.77

    $11,162.31 / month · 6,307.2M output tokens / month

    System B

    Cost / million accepted output tokens

    $1.87

    $7,367.92 / month · 3,942M output tokens / month

    B matches A's current cost per token at 63.5% utilization. This compares unit costs, not equal capacity or guaranteed demand.

    Calculation and exclusions

    Monthly cost = (purchase + setup) ÷ (years × 12) + 730 × PUE × electricity price × [active kW × load + idle kW × (1 − load)] + monthly operations. Monthly output = goodput × 730 × 3,600 × load. Load is utilization ÷ 100. Both systems remain powered on all month; depreciation is straight-line with zero residual value.

    Count only accepted output, after accounting for prefill, queueing, retries and quality failures. Add redundancy, networking, licensing and staff to your quote or operating cost. Tax, financing, foreign exchange and resale are excluded. Do not charge cooling twice if colocation pricing already includes it. At low demand, renting briefly may cost less than owning idle hardware.

    Does the NPU market make sense?

    Yes, for repeatable inference with proven economics. The case weakens when the workload changes faster than the software can support it, or the machine sits idle. A lower chip power rating is only one input.

    The case for a specialist accelerator

    • A stable model family and operator set, with enough recurring demand to keep capacity useful.
    • A measured cost advantage after porting, support, redundancy and staff; equal output quality and tail latency.
    • Power- or cooling-constrained sites where usable tokens per rack matter, and an operational fallback exists.

    When GPUs, Mac or an API fit better

    • Frequent model changes, unsupported operators, training or custom kernels: flexibility can outweigh a power advantage.
    • Small or bursty demand: cloud/API spending can follow use; local Mac may be enough for private, low-concurrency work.
    • No comparable throughput or binding total quote: run a pilot before treating projected savings as a business case.

    Our assessment: SKT and Upstage cases show a commercial role for Korean NPUs, not universal GPU replacement. Falling model API prices and improving GPU software also move the break-even point. For an AI app business, revenue growth brings inference expense; judge contribution margin per retained customer, not token volume alone.

    Rebellions / SKT deployment · 2026-09-21 ↗Furiosa / Upstage production case · September 2026 ↗

    Ask every vendor for the same acceptance test

    Pin the model revision, quantization and quality test; prompt/output length distribution; concurrency; p95/p99 time-to-first-token and time-per-output-token; accepted output tok/s; measured whole-server wall power; runtime version; run duration; and failure/retry rate. Include a long-context case, a small-model case and the largest model you actually need. Request a reproducible log and an itemized three-year quote.

    Explore open-weight models in Work →