Sampling Thinking Machines’ Inkling and the Alias Pattern

In July I wrote The Model Sheet — how a filename like Qwen2.5-Coder-32B-Instruct-Q5_K_M.gguf breaks into family, domain, params, training stage, quantization, and format. Those fields predict behavior before you download. What comes after the spec sheet is the trial: load the weights, run a real prompt, see if the model behaves as advertised.

Lately that trial has been Thinking Machines' Inkling: 975B total / 41B active MoE, Apache 2.0, 1M context. This post covers the mechanics of running it under Claude Code — a thin wrapper, one environment variable that redirects the CLI, and the two routes to the same weights (Baseten direct, Vercel AI Gateway) that behave very differently under a coding agent's traffic.

Inkling is the sample here. My current split is simpler: DeepSeek V4 Flash 0731 runs locally through ds4 for truly private or sensitive work, at 30 to 40 tokens per second on my M5 Max. Most shareable work goes through the Vercel AI Gateway, usually back to DeepSeek V4, while claude-gateway <model-name> is how I keep trying open-weight and other models without rebuilding my toolchain. The measured case for that setup is in my Vercel AI Gateway review.

The wrapper works because the serving side speaks Anthropic's Messages API. Claude Code talks /v1/messages — system / user / assistant roles, content blocks, tool calls. llama.cpp ships a native /v1/messages endpoint now, ds4-server has one, the Vercel AI Gateway exposes an Anthropic-compatible endpoint, and Baseten's direct inference endpoint does too. Point ANTHROPIC_BASE_URL at any of them and you get a Claude Code that thinks it's talking to Claude. No proxy, no translation layer. The shims (LiteLLM, router processes) are only needed for providers that stop at OpenAI compatibility — Groq and Cerebras are in that camp, which is one real reason they aren't in my rotation despite their speed.

The route that looked perfect

Baseten's direct endpoint (https://inference.baseten.co) exposes that native /v1/messages route over plain HTTPS. The key lives at ~/.config/baseten/key with mode 0600; the wrapper reads it (or $BASETEN_API_KEY if set), exports the endpoint and the model identifier, then executes claude. INKLING_VIA=baseten selects the direct route; INKLING_SMALL=1 swaps in inkling-small when the full model is overkill.

The direct route mattered for this trial because it cut out a middleman. The Vercel AI Gateway pools requests through a single bill with provider failover — useful, but it adds a hop and a config surface. Direct Baseten was one less dependency between the CLI I already live in and the weights that were actually answering. That was the plan.

One mechanical detail worth spelling out, because it's the difference between a wrapper that works and one that half-works: Claude Code makes several kinds of model calls. The main loop runs one model, background tasks like title and summary generation default to a haiku-class model, and subagents can be configured to run a different model again. Any of those defaults that leaks through arrives at the provider as an unknown Anthropic model name and errors. So the wrapper pins every tier to the one hosted model:

export ANTHROPIC_BASE_URL="https://inference.baseten.co"
export ANTHROPIC_MODEL="$MODEL"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="$MODEL"
export ANTHROPIC_DEFAULT_SONNET_MODEL="$MODEL"
export ANTHROPIC_DEFAULT_OPUS_MODEL="$MODEL"
export CLAUDE_CODE_SUBAGENT_MODEL="$MODEL"

With that in place, the direct route survived exactly one trial before the rate limiter found it.

The rate limit that killed the direct route

Baseten's Basic tier shares a 100k tokens/min pool across the workspace. Verified workspaces skip the cap; mine wasn't verified yet. Here is why that pool dies on contact with a coding agent, and the numbers are the agent's shape, not Inkling's. I measured it in The Price Floor on a different model: over the 12.3-day sample there, 4,127 assistant messages consumed just over a billion input tokens — 96.6% of them prompt-cache reads, because an agent re-sends its growing context on every call: system prompt, tool definitions, file contents, prior tool results. That averages out to roughly 266k input tokens per assistant message. Against a provider with cache-aware limits, nearly all of that is a cheap cache hit, but the pool counts tokens, not their price. However you meter it, a turn that carries a quarter-million input tokens does not fit inside a shared 100k/min pool — whatever model is answering.

The failure mode is deterministic and ugly. A turn that streams a big codebase hits the ceiling mid-answer, the request 429s, and Claude Code retries — which burns more of the pool on prompts that came back empty. The interrupt happens during work, so the loop never converges on anything. The first time it struck I watched a straightforward edit collapse into a series of rate-limit headers and an unfinished file.

The fix was temporary on purpose: switch the wrapper back to the Gateway to finish the task, and keep the direct path staged behind workspace verification. The Gateway absorbs the burst and fails over between providers, so the interruption cost about one command. The distinction I want to keep crisp is that the rate cap is a workspace property, not a model quirk. Inkling is not slow; the shared Basic tier is simply sized for request/response traffic, and an agent is a context firehose. Once the workspace is verified — or a dedicated deployment is provisioned with a private key — the direct route is the stable path and the Gateway becomes the fallback.

Why the Gateway leg keeps prompt caching on

The local wrappers (claude-laguna, claude-deepseek) export DISABLE_PROMPT_CACHING=1, because llama-server and ds4 manage their own KV cache and client-side cache-control headers are noise to them. The Gateway wrapper does the opposite, and the script says why in a comment I wrote so I wouldn't "clean it up" later:

# NOTE: prompt caching is deliberately LEFT ON here — unlike the local
# `claude-deepseek` wrapper, which disables it because ds4 caches itself.
# ~96% of this workload's input tokens are cache reads, and the Gateway
# bills those at $0.0028/M vs $0.09/M uncached. Disabling caching would
# cost roughly 30x more. Do not add DISABLE_PROMPT_CACHING here.

The measured shape of an agent workload is context-dominated: in my sample, 218 input tokens for every output token. At that ratio the cache-hit discount dominates the headline per-token price, which is why the same flag is correct in one wrapper and a 30× mistake in another. The alias pattern makes this legible — each wrapper is a place to write down the tuning its route needs.

The alias: claude-inkling

The whole wrapper is short enough to read:

#!/usr/bin/env bash
# Claude Code on Thinking Machines' Inkling (975B/41B-active MoE, Apache 2.0,
# 1M context).
#
# Default route: Vercel AI Gateway (thinkingmachines/inkling) — one bill,
# provider failover, and none of Baseten's Basic-tier 100k tokens/min cap,
# which a single Claude Code turn exhausts on its own.
#
#   INKLING_SMALL=1  claude-inkling    -> inkling-small via Gateway
#   INKLING_VIA=baseten claude-inkling -> Baseten direct (/v1/messages native;
#       useful once the workspace is verified or on a dedicated deployment;
#       key from $BASETEN_API_KEY or ~/.config/baseten/key)
set -euo pipefail

MODEL="thinkingmachines/inkling"
[ "${INKLING_SMALL:-0}" = "1" ] && MODEL="thinkingmachines/inkling-small"

if [ "${INKLING_VIA:-gateway}" = "baseten" ]; then
  KEY="${BASETEN_API_KEY:-}"
  [ -z "$KEY" ] && [ -r "$HOME/.config/baseten/key" ] && KEY="$(cat "$HOME/.config/baseten/key")"
  [ -n "$KEY" ] || { echo "claude-inkling: no Baseten key" >&2; exit 1; }
  export ANTHROPIC_BASE_URL="https://inference.baseten.co"
  export ANTHROPIC_API_KEY="$KEY"
  export ANTHROPIC_AUTH_TOKEN="$KEY"
  export ANTHROPIC_MODEL="$MODEL"
  export ANTHROPIC_DEFAULT_HAIKU_MODEL="$MODEL"
  export ANTHROPIC_DEFAULT_SONNET_MODEL="$MODEL"
  export ANTHROPIC_DEFAULT_OPUS_MODEL="$MODEL"
  export CLAUDE_CODE_SUBAGENT_MODEL="$MODEL"
  export CLAUDE_CODE_MAX_OUTPUT_TOKENS="${CLAUDE_CODE_MAX_OUTPUT_TOKENS:-32000}"
  export DISABLE_TELEMETRY=1
  export DISABLE_NON_ESSENTIAL_MODEL_CALLS=1
  export DISABLE_ERROR_REPORTING=1
  exec claude --dangerously-skip-permissions "$@"
fi

exec claude-gateway "$MODEL" "$@"

The Gateway path delegates to claude-gateway, a shared implementation that takes the model id as its first argument. The names describe jobs in my current routing, not a permanent leaderboard:

• claude-deepseek — DeepSeek V4 Flash 0731 locally through ds4's custom engine at 127.0.0.1:8087. It generates at 30 to 40 tok/s on my M5 Max and is the route for truly private or sensitive work. DEEPSEEK_THINK=0 flips the server to its non-thinking alias for cheaper tool loops.
• claude-flash — deepseek/deepseek-v4-flash-0731 through the Vercel AI Gateway. This is the everyday route for most work that can leave the machine: roughly 73 tok/s in my trials, the cheapest capable option in my measured sample, and multiple provider deployments behind one endpoint.
• claude-gateway <model-name> — the audition room. Any provider/model ID in Vercel's catalog can replace DeepSeek for a session, which is how I keep sampling open-weight and other models and right-sizing the route without collecting provider accounts.
• claude-pro — DeepSeek V4 Pro through the Gateway. Roughly 3× Flash's token price; for harder problems, not a default.
• claude-inkling — Inkling through the Gateway by default, with INKLING_VIA=baseten available for the direct experiment documented here.
• claude-laguna — local Laguna-S-2.1 on llama.cpp at 127.0.0.1:8086, still useful offline but no longer the local default.

One alias per model, one endpoint per alias, zero proxy overhead. A few conventions repeat across all of them, and they're the boring 80% of why the pattern holds up:

• Keys are files, not shell history. Each hosted key lives at ~/.config/<provider>/key, mode 0600, with an env var override for CI or one-offs. No keys in .zshrc, nothing to leak in a pasted transcript.
• Preflight before launch. Every wrapper curls the endpoint's /v1/models before executing claude — 3s timeout for localhost, 8s for the Gateway. A dead local server or an expired key fails at the shell with the command that fixes it (start it with 'laguna' first), instead of three prompts into a session. The Gateway wrapper goes one step further and checks the requested model id against the catalog it just fetched, printing near-matches when it isn't there — so a typo'd id fails at launch with suggestions instead of erroring mid-session.
• Background calls are pinned or disabled. DISABLE_NON_ESSENTIAL_MODEL_CALLS=1 everywhere, because burning local decode time — or a metered pool — on conversation-title generation is pure waste.
• A 32k output budget by default, for the reasoning-model failure mode below.

Because every endpoint speaks the Anthropic Messages API, the client sends {"model":"thinkingmachines/inkling","messages":[...]} and gets back a standard message response with content blocks. The alias just bridges Anthropic's CLI naming convention and the provider's model registry.

What the trial measures

A sample is a short, representative task from real work — a coding snippet, a reasoning chain, a format conversion — scored pass or fail against a known answer. The model either produces working output or it doesn't. Binary. I keep the prompt set in a plain file under version control, per the procedure at the end of The Model Sheet: define the task, shortlist by spec, quantize (or route) to fit, test on your own work. The next release has to earn the swap.

Inkling's behavior so far aligns with its MoE spec: 41B active out of 975B total. Fast prefill, slower decode (bandwidth-bound, like any large model), and a generous output budget needed before visible answers appear. That last one is the standard reasoning-model gotcha: thinking models spend output tokens on a hidden reasoning span before the reply, and if max_tokens is small the whole budget goes to thinking and the content field comes back empty. It looks like a broken endpoint and it's actually a config default. The 32k budget in the wrappers exists because this has bitten me on more than one model.

This connects back to The Benchmark: a leaderboard rank collapses too many dimensions into one number. The fields from The Model Sheet — dense vs MoE, active params, quantization level, training stage — are the actual variables, and the operational fields this post adds (rate-cap tier, cache-hit pricing, protocol compatibility) belong on the same sheet. The alias wrapper just makes all of them accessible to the CLI I already use.

Why I wrote this one by hand (with some help)

Here's the honest part. After all the endpoint fiddling and image regeneration, Inkling was not strong enough to handle this post. It did fine on isolated turns — a state change here, a format conversion there — but across the full arc of a long, personal, heavily-revised blog post, it lost the thread. It produced drafts that hit the house style on the surface and missed it in the connective tissue: the honest pacing, the measured failure framing, the place where a paragraph about my own tooling needs to sound like me and not like a model reciting a design doc.

So I swapped to DeepSeek V4 Flash 0731 through the Vercel AI Gateway to get it over the line. This outcome is the data point the trial is supposed to produce, recorded next to the pass/fail samples. Inkling handled bounded coding turns and lost the thread on a long, first-person, style-constrained piece; both facts belong in the same session log. The alias wrapper made the swap a one-line change: same CLI, different endpoint. That is what the wrapper is for, and the broader browse-pick-run-measure loop is the center of my Vercel AI Gateway review.

Where this fits

The routing boundary is data sensitivity. claude-deepseek serves DeepSeek V4 Flash 0731 locally through ds4 when nothing can leave the laptop. It is fast enough at 30 to 40 t/s that privacy no longer means waiting on an unusably slow model. For most work, claude-flash sends the same model family through Vercel's Gateway. When I want to learn the current model field, the general claude-gateway <model-name> alias lets me swap in a candidate from Vercel's model table, where cost sits next to latency and throughput, and run the comparison while the shortlist is still fresh.

Inkling earned a useful sample and a durable alias. Daily work stays centered on DeepSeek V4, split between local ds4 and the Gateway according to data sensitivity. The next model will get the same treatment. A named wrapper is the record of a job a model has earned, and the general Gateway alias is how models audition for one.