The Price Floor

I'm an AI engineer. I work on an applied AI team during the day, and I build things for myself constantly outside of it. I recently bought an M5 Max MacBook Pro with 128 GB of unified memory so I could get serious about local inference, meaning running language models on my own machine instead of calling someone's API. On July 31 DeepSeek gave the new machine its first real assignment: V4 Flash 0731, a retrained checkpoint of their 284-billion-parameter open-weight model that outperforms their own flagship preview on agent benchmarks, published under the permissive MIT license^[1]. Open weights means the trained parameters are downloadable; a checkpoint is a snapshot of those parameters at a point in training, and this one differs from April's preview only in post-training, the finishing phase that shapes a pretrained model's behavior.

I wanted to answer one question with my own measurements: what does it cost to run a frontier-class open model today, in dollars and in seconds? The answer took a full day and touched every layer of the stack. The same weights ran five times faster after swapping inference engines on identical hardware. A speculative decoder worth 1.4x on NVIDIA silicon measured at zero on Apple's. Two cloud providers priced 5x apart produced statistically identical throughput. Three vendors selling genuinely fast inference turned out to have nothing for sale, and the price that dominates an agent workload's bill turned out to be one that most provider comparisons never print.

This drawing walks through those measurements in order, and it explains each piece of technology as it appears, because half the value of the exercise was learning what these components actually do. If you know what a KV cache is, the explanations are skimmable. If you don't, they're the point.

Two units run through everything that follows. A token is the chunk models actually read and write, about three-quarters of an English word on average, and throughput is measured in tokens per second. For scale, a person reading at 250 words per minute consumes text at roughly 5.5 tokens per second, so 40 tok/s is several times faster than you read, and 300 tok/s fills a screen faster than you can scroll it. An agent, the workload this drawing prices, is a model wired into a loop by a harness: a program that feeds the model your files and conversation, lets it call tools like "edit this file" or "run this command," and executes what it asks. Coding assistants such as Claude Code and Codex are agents in this sense.

V4 Flash 0731 is a mixture-of-experts model. MoE is worth understanding before any of the numbers make sense: instead of one dense stack of weights that every token passes through, the model holds many alternative feed-forward blocks, called experts, and a small router picks a few of them per token. V4 Flash has 284 billion parameters in total but activates only 13 billion per token. The total count sets how much memory the model occupies. The active count sets how much work each token costs, which is why a 284B MoE can generate faster than a 70B dense model. It also has a 1-million-token context window, the running transcript of everything the model can see at once, which at a million tokens holds roughly a large codebase. "Frontier-class" throughout this drawing means within sight of the best proprietary models on hard tasks, which is where DeepSeek's published agent benchmarks place this checkpoint.

Open weights make it a controlled experiment. The same model can run on my desk, on a rented GPU, or behind half a dozen companies' APIs, so the model stays constant while engine, silicon, provider, and price vary one at a time.

Three claims were on the table when I started. Published benchmarks said the model generates around 107 tokens per second on DeepSeek's own API and 267 on the fastest reseller^[5]. Local-inference blogs said a 128 GB Mac runs it in the twenties. And every provider pricing page implied that fast inference is a commodity you buy with a credit card. All three failed measurement by the end of the day.

Some vocabulary for this sheet. An inference engine is the software that loads a model's weights and schedules its math onto a chip; llama.cpp is the de facto standard engine for running open models on consumer hardware, and GGUF is its weight format. Quantization stores weights at lower precision to shrink them: DeepSeek ships V4 Flash at a mixed 4-bit and 8-bit floating-point precision totaling about 170 GB, and a further 2-bit quantization compresses that to 97 GB, small enough to fit in this laptop's memory with room left to work. Apple's unified memory is what makes that sentence possible on a laptop at all: the CPU and GPU share one 128 GB pool, so the whole model sits GPU-addressable without a discrete graphics card.

Mainline llama.cpp merged V4 support in June^[2], so that was the first run: the 2-bit build on the stock engine. It loads, the output is correct, and it generates 6.05 tokens per second. The interesting part is that the PR's own test hardware, a completely different machine, also reported about 6. When two unrelated systems land on the same throughput, the shared bottleneck is the software.

The expected ceiling is set by physics. Generating a token on a memory-bound MoE model means moving the active parameters through the memory bus once. Thirteen billion active parameters at roughly 2.7 bits per weight is about 4.4 GB per token, and the M5 Max moves 614 GB/s:

Mainline llama.cpp reaches 4% of that bound. The same weights in ds4, a dedicated engine antirez wrote for exactly this model family^[3], reach 30 to 40 tokens per second on the same GPU, with prompt processing between 377 and 448 tokens per second and an 8-second model load. The difference is kernels: the small GPU programs an engine dispatches to do the actual matrix math, written against a GPU platform such as NVIDIA's CUDA or Apple's Metal. V4 introduced new attention machinery, a lightning indexer and compressed-attention projections, and mainline's first implementation of those operations is correct but unoptimized. The PR says so directly: architecture support first, optimization later. ds4 specializes in this one architecture and pays for that narrowness with a 5.2x speedup.

The practical rule: when a new architecture lands, the first working implementation tells you nothing about what the hardware can do, and any buy-versus-build decision made against it inherits the error.

The day's cleanest negative result came from speculative decoding. The technique pairs the big model with a small draft model that guesses several future tokens; the big model verifies the guesses in a single pass, and every accepted guess advances the stream by more than one token per step. DeepSeek publishes a draft module for V4 Flash called DSpark, and an optimized CUDA fork reports 1.38x mean speedup from it, 1.71x on code^[4]. On my Metal GPU, with the drafter confirmed loaded: 39.83 tokens per second without it, 39.99 with. The gain rounds to zero. Speculative decoding only pays when the verification pass is cheap relative to plain decoding, and that depends on kernel-level optimization the Metal path doesn't have. A published speedup is a property of the full software stack it was measured on, and it does not transfer to a different one.

Serving a language model has two phases, and keeping them separate is the key to reading any benchmark. Prefill is the model ingesting your prompt: compute-heavy, highly parallel, measured in how many prompt tokens per second it can absorb. Decode is the model generating output one token at a time, each step re-reading the active weights, which is why decode speed tracks memory bandwidth. A number quoted without naming its phase is close to meaningless.

The DGX Spark makes a good case study. It's NVIDIA's compact desktop AI machine, a GB10 chip with 128 GB of memory shared between CPU and GPU, positioned for exactly the kind of local-model work this drawing is about. On paper it looks weaker than this laptop for decode: its memory moves 273 GB/s against the Mac's 614, and decode is bandwidth-bound. The upstream ds4 README confirms that reading, benchmarking the Spark at 13.75 tokens per second, less than half the Mac. I quoted that number for two weeks as a reason not to buy one.

The fuller picture arrived with a CUDA fork of ds4 whose kernels were rebuilt for the chip^[4]. On the same hardware it reports prefill around 1,000 tokens per second at a 12k-token prompt, 2.5x the Mac. Plain decode lands at 18 to 20 tokens per second, which sounds unimpressive until you notice it's 90% of the chip's theoretical bandwidth ceiling, meaning the software is done improving and the silicon is the limit. It also serves 59 tokens per second aggregate across twelve concurrent requests, and warm starts 7x faster by reusing previously computed prefixes.

That last set of numbers introduces one more concept: continuous batching. An engine that batches serves many requests inside each model pass, so total throughput across all streams (aggregate) can be several times what any single stream sees. The Mac path has no batching at all; the Spark fork batches up to 128 requests. Whether that matters depends entirely on whether you run one session or many.

So the same two machines produce opposite verdicts depending on the metric. The Mac wins single-stream decode. The Spark wins prefill, concurrency, and warm-start latency, which between them cover most of what an agent workload does all day. Neither benchmark lies; they measure different phases. The error was mine, in quoting one phase as if it were the whole answer.

Open weights create a resale market: because anyone can download the model, DeepSeek's own first-party API competes with resellers hosting the identical checkpoint on their own GPUs. The published table showed first-party at 107 tokens per second and the fastest reseller at 267^[5]. I benchmarked two endpoints serving the identical 0731 checkpoint. One was Fireworks, a GPU inference provider that hosts open models behind an API. The other was Vercel's AI Gateway, a router that fronts eight different providers behind one endpoint and one bill. Same prompt, five trials each, the same evening, with responses streamed, meaning tokens arrive one by one as generated and the clock runs from request to final token.

Fireworks came back at a median of 72 tokens per second end-to-end, the Gateway at 73. Both sit 30 to 45% below the published figures. The larger finding was variance: individual runs ranged from 20 to 99 tokens per second, a wider spread than any difference between the providers. Time-to-first-token, the wait before the first word of a response appears, told a similar story. Fireworks produced outliers of 8 and 9 seconds on otherwise sub-300ms starts, while the Gateway held between 0.35 and 0.63 seconds on every trial. For interactive use, that tail latency shapes the experience more than the median throughput does.

The two endpoints do differ on one thing: price. Fireworks charges $0.028 per million cached input tokens where the Gateway and DeepSeek's own API charge $0.0028^[6], a 10x difference on a line item that the next sheet shows to be most of an agent workload's bill, attached to no measurable difference in throughput. I also measured Groq, a provider built on custom silicon rather than GPUs, for reference on what fast currently means: 316 tokens per second on gpt-oss-120b and 372 on Llama 3.3 70B, with first tokens inside three-quarters of a second. Those are smaller models than V4 Flash, and Sheet 6 covers why that trade is currently unavoidable.

This sheet needs a disclosure up front, because its most useful number comes from my weakest data.

The mechanism is structural, so it comes first. A coding agent re-sends its entire context on every turn: the system prompt, the tool definitions, the whole conversation so far. Providers exploit this with prompt caching. The server stores the processed form of a prompt prefix, and when the next request starts with the same bytes, it skips the recomputation and bills those tokens at a discounted cache-hit rate instead of the full input rate. None of this depends on who you are or how much you run. It follows from how agent harnesses are built, and it means that for any long-lived-context workload, the cache-hit price is the price.

Now the data, honestly labeled. This laptop is weeks old. Its 12-day ledger of 4,127 assistant turns is mostly the exploration this drawing documents, a sliver of my actual working life. My working days run several concurrent agent sessions against frontier APIs across multiple codebases, and none of that traffic appears in this sample. So the ledger's volume says nothing about how I work, and I won't present it as if it did. What it can show is composition. Of everything the sample sent upstream, 1.065 billion input tokens were cache reads, against 33 million cache writes and 183 thousand fresh input tokens, with 5 million tokens generated. Cache reads were 96.6% of all input, and input outnumbered output 218 to 1. That fingerprint comes from the harness, and the harness is the same one running my heavier sessions at work.

Vendors price cache hits with different philosophies. DeepSeek charges 2% of the miss price^[6]. Anthropic charges 10% of the input price. Groq discounts hits by exactly half^[7]. Run the sample ledger through each rate card and the same tokens produce bills two orders of magnitude apart:

The habit worth taking away: before comparing providers, pull your own usage records and compute your own h. If you run agents over real codebases it will be high, and then the cache-hit line is the price you're shopping. The rest of the rate card barely participates in the total.

The plan after benchmarking was to pick a fast-silicon vendor, add a payment method, and get 300+ tokens per second for daily work. Three vendors, three closed doors.

Makora, the reseller behind the 267 tok/s listing, shows both of its self-serve plans, $20 and $200 a month, marked sold out. Cerebras Code Pro at $50 and Code Max at $200, flat-rate coding subscriptions with daily token allowances, are both sold out^[8], and the restocks since the plans launched in May have sold out as well. Groq's paid Developer tier has been closed to upgrades for months; their community forum carries threads titled "When is developer tier coming back?" without an answer. Groq's free tier remains open but is capped at 6,000 tokens per minute, and a single agent-session context exceeds that minute.

The economics of the pattern are straightforward. Cerebras Code Max sells 120 million tokens a day, 3.6 billion a month, for $200. At even a budget metered rate, that allowance is worth roughly $1,080. The flat tiers price tokens below metered retail, which only works if the seats are capped, and the caps are full. The metered pay-per-token APIs at these same companies remain open, because a metered customer is profitable at any volume. What is actually scarce in August 2026 is fast inference at retail prices.

The scarcity has a physical explanation worth knowing. Groq builds LPUs, language processing units, custom chips designed only for model serving. Cerebras builds wafer-scale engines, single processors the size of an entire silicon wafer rather than a chip cut from one. Both get their speed the same way: they hold the model's weights in SRAM, the memory fabricated directly on the processor itself. SRAM is orders of magnitude faster than the DRAM chips that sit next to a GPU, which is the point, but it's available only in gigabytes where DRAM comes in hundreds of them, which is the catch. Speed from SRAM comes with a hard cap on model size. The catalog these vendors serve at 300 to 2,100 tokens per second consists of models like gpt-oss-120b and Llama 70B, a full tier below a 284B-parameter MoE. As of this writing, you can buy frontier-class quality at about 73 tokens per second, or a smaller model at 372, and no vendor sells both on one invoice.

If no vendor sells fast inference on a frontier-class open model, the remaining route is owning the hardware. The reference build in mid-2026 uses four RTX PRO 6000 Blackwell cards, NVIDIA's current workstation flagship, each carrying 96 GB of GDDR7, the current generation of the dedicated high-speed memory that lives on a graphics card, at roughly $34k all-in for the machine. The combined 384 GB fits V4 Flash at the precision DeepSeek ships natively, about 170 GB of weights with no quantization loss, and leaves 210 GB for the KV cache and concurrent sessions. The KV cache is the attention state a model keeps for its context; it grows with context length and with the number of simultaneous sessions, so the headroom is what determines how many agents the box can serve at once.

Two published data points bracket what the build delivers. Under one serving stack, two cards reach 210 tokens per second single-stream and around 700 aggregate. Under another stack, the same silicon benchmarks as low as 25. That spread is Sheet 2's engine tax again, at 34-thousand-dollar stakes, and it is the largest risk in the purchase. The second risk is timing: NVIDIA's list price on the card rose 55% in sixteen months, from $8,565 to $13,250, on a GDDR7 supply shortage^[9], and street prices swing thousands between retailers.

Amortized over three years with power included:

Above $1,160 a month of equivalent API spend, the rig wins. Below it, the API does. Which comparison brings this drawing to the mistake most worth documenting. I first ran that crossover against the 12-day ledger from Sheet 5, the $17-a-month one, and concluded the rig was sixty times too expensive. The arithmetic was correct and the conclusion was wrong, for the reason Sheet 5 disclosed: that ledger is a new personal machine's first days. The workload a rig would actually absorb is the office one, concurrent frontier-API sessions that cost hundreds of dollars a day, and against that workload the crossover clears in the first week. A precisely measured but unrepresentative sample produced a worse decision than no measurement, because the precision made it convincing.

The larger V4 Pro model stays out of home reach in any configuration: 464 GB at a 2-bit quantization means five cards before any headroom, and native precision means nine. The home floor is a Flash-class floor. My own version of this plan now has a date rather than a parts list, because every number in this sheet will have moved before the power circuit goes in.

The day ended without a single winner, because each measurement pointed at a different tool for a different slice of work.

The laptop holds the private slice. It's the only option with zero data egress, and 30 to 40 tokens per second is workable for anything that can't leave the machine. The constraint is real: the cheapest capable API stores data on servers in China under terms with no training opt-out, which is fine for open-source work and disqualifying for anything proprietary. Metered V4 Flash through the Gateway holds the default slice at pennies for anything shareable. Fast silicon, when a seat opens, fits iteration work, the boilerplate and refactors where a wrong answer is cheap and a fast one compounds, on a smaller model that is good enough there. The frontier API keeps the problems worth $25 per million output tokens. The rig waits for the month the spend justifies the floor.

Every number here has a short shelf life. Serving stacks will close some of the gap to EQ. 2.1's bound, GDDR7 supply will loosen or it won't, and a vendor may reopen a flat-rate fast tier. The method is what keeps: pull your own usage ledger, weight each provider's prices by your own token composition, and time published throughput figures with your own clock before routing money by them. That method was the new laptop's first real lesson.

For the mechanics underneath these measurements, the series has working drawings on how inference engines schedule a forward pass, what benchmark numbers do and don't establish, and a local speech pipeline built under the same constraints.