The Voice Box: Running Your Own ElevenLabs, Offline and Free

I wanted to type a sentence and hear it spoken locally. Paid APIs have done that for years. A paid API sends the text over a network and charges by the character. The vendor also decides what it will synthesize. Now the same job runs on a laptop. The gap between the two closed while I wasn't paying attention.

This drawing follows the complete build: a 5-billion-parameter open-weight speech model^[1], an Apple Silicon runtime, a preset voice library I had to make because the model ships without one, and a small web app that turns arbitrary text into a 44.1 kHz waveform in roughly real time. I measured every number on my own machine, an M5 Max with 128 GB of unified memory. I've also kept the two bugs that cost me the most time because both came from undocumented runtime behavior.

The model has no built-in voices. It is a pure zero-shot cloner, and without a reference it produces a different person every single time. A selectable voice library took half the project to build.

My workflow is ordinary: draft a line, hear it, change a word, and hear it again. I may run twenty iterations on a single sentence to get the emphasis right. Hosted synthesis charges for every pass because its meter runs on output characters.

I had three objections beyond the bill.

Every draft leaves the machine. I can accept that for a demo script. Work under an NDA and personal material should stay local, especially when I haven't decided whether to publish it. I don't want to send such drafts to a third party, where they may be logged, twenty times an hour.

The vendor decides what the model will say and can change its endpoint policy between Tuesday and Thursday. The same policy covers a security researcher writing phishing-awareness training material and a bad actor. I want the filter to be my judgment, applied by me, to my own work.

Queue and network latency also make response time unpredictable. A local model that takes two seconds beats a remote model that takes one second plus a variable network wait. I know how long the local request will take, and it still works on a plane.

Speech synthesis has followed the same curve as text models, roughly two years behind and with far less commentary. A lab publishes a proprietary flagship, then distills or releases a smaller sibling with open weights. Within a couple of quarters, the open sibling can do what the flagship did a year earlier on hardware you already own.

Fish Audio followed that pattern. Its S1 generation paired a proprietary 4B model with a 0.5B distilled version under CC-BY-NC-SA^[3]. The smaller model covers 13 languages and runs on modest hardware. It accepts inline emotional markers and supports zero-shot cloning. The S2 generation followed in March 2026 with a technical report^[1] and downloadable weights for S2 Pro^[2]. It uses a dual-autoregressive design and covers 80+ languages. Training used more than 10 million hours. Inline natural-language tags provide word-level delivery control.

The architecture explains the performance numbers later. Generation splits across two autoregressive stacks: a 4B "slow" model handles what is said and how it should land, while a 400M "fast" model expands those semantic decisions into acoustic detail. The acoustic tokens then pass through a separate neural codec to produce a waveform. The expensive work runs at semantic-step frequency, which lets a 5B model hold real-time on a laptop.

I did not run a head-to-head listening test against ElevenLabs. Hosted vendors still have better voice libraries and a more polished product with support. The two features I paid for were expressive synthesis with fine control and cloning a voice from a short sample. Both work here, on my hardware, for the cost of the electricity, which covers my iteration-heavy drafting.

S2 Pro uses the Fish Audio Research License: research and non-commercial use free, commercial use requires a separate agreement. The 0.5B S1-mini is CC-BY-NC-SA-4.0. The weights do not enforce either license. A paid product using either model still requires a licensing decision.

The reference implementation^[4] uses an SGLang inference engine aimed at datacenter NVIDIA hardware. On Apple Silicon, it executes too slowly for this use. Reports of 25 to 70 seconds for a single short sentence rule it out for the iteration loop I described in Sheet 1.

I used Apple's MLX array framework^[5] through mlx-audio^[6], which has a native S2 Pro backend. Ready-made conversions of the weights are published^[7], so setup takes a virtual environment and one download.

uv venv --python 3.12
uv pip install mlx-audio        # pulls mlx, mlx-lm, transformers

The mlx-audio package names this model's module fish_qwen3_omni; the name contains neither "s2" nor "pro". A grep for either obvious term finds nothing.

The first generation used inline tags and produced 12.07 seconds of audio in 14.7 seconds of wall time. That is a real-time factor of 0.82, with memory peaking at 17.2 GB. On the second run, the weights came from cache and the model was resident in 1.1 seconds. That startup time makes the interactive app usable.

Real-time factor above 1.0 means audio is produced faster than it plays. An RTF of roughly 1.0 across the range makes a ten-second line take about ten seconds. I can rewrite a sentence and hear the result without breaking the editing loop.

I next tried to populate the voice selector. S2 Pro is a pure zero-shot cloner with no speaker embeddings or named voices. I confirmed the lack of a built-in voice library three ways. The tokenizer's special tokens are modality and codec markers such as <|voice|> and <|semantic:N|>; none resembles a speaker name. The CLI accepts a --voice flag, prints it back, and changes nothing. The mlx-audio Fish backend shows why on one line:

del voice, repetition_penalty, verbose, kwargs

The backend accepts the argument and discards it, so passing --voice Chelsie is exactly equivalent to passing nothing.

Without a reference, the model invents a speaker. Two generations of the same sentence produced median fundamental frequencies of 205.1 Hz and 167.7 Hz and sounded like different people. Four runs of one sentence spanned 21.2 Hz.

A reference clip anchors the speaker identity. I fed one generation back as its own reference and measured 198.2 Hz and 198.6 Hz across two runs. Across four presets with two runs each, the mean within-preset spread was 5.1 Hz against 21.2 Hz unpinned, roughly four times tighter.

A reference narrows the variation without eliminating it. Two runs of the same preset still differ by a few Hz, and one preset drifted 10.7. I regenerate takes that wander.

I store preset references in a folder. Any clip can work, including one the model generated. I measured a batch of randomly rolled speakers and saved the distinct ones for reuse.

I used pitch as a cheap proxy so the selection could run automatically. An autocorrelation estimator over voiced frames gives each clip a median fundamental frequency, enough to tell two candidates apart and sort them. The measurement says little else about a voice.

The first batch of ten rolls clustered between 161 and 192 Hz and yielded three usable presets. The model's default sampling sits in a narrow band. Two changes widened it: raising temperature to 0.95, and varying the seed sentence, because the speaker the model invents correlates with the content it reads. A second batch of sixteen spanned 148 to 223 Hz and added four more.

My first labels used each reference clip's pitch. Probe runs showed a systematic downward shift: the reference measuring 222.7 Hz generated speech at 183.1 Hz, and the 212 Hz reference produced 201.8 Hz. A reference's own pitch overstates its clone by 10 to 40 Hz.

I changed the calibration pass to probe every voice with a fixed sentence and name the preset from the measured output, so the label predicts what I will hear.

Seven presets came out of it, spanning 130 to 202 Hz by measured output. They cluster in the middle because that is where the model's unpinned distribution sits. A real reference clip, with a 5 to 15 second sample and its exact transcript, bypasses this process and clones a specific person.

The exact transcript is required, and omitting it is the single most common cloning failure. The model conditions on the audio-text pairing and produces nothing useful from reference audio alone.

I used the command line to prove the model worked, but it did not fit the actual task. Every invocation reloaded the weights. Voice selection required remembering a filename and pasting a transcript.

I replaced it with a local web app. The HTTP layer fits in one Python file and uses only the standard library. It holds the model resident while serving a single page on loopback. The page has a text box and voice dropdown, plus delivery tags, sampling controls, an inline player, and a button that promotes any generation I like into the preset library.

I had to fix two bugs caused by the runtime.

Decode reference audio before calling the model. The library's own README passes a filename to ref_audio. A direct call with that filename fails deep inside the model with 'str' object has no attribute 'ndim'. The CLI decodes the file first and passes an array. Direct model calls need the same decoded samples:

from mlx_audio.utils import load_audio
kwargs["ref_audio"] = load_audio(str(ref_path), sample_rate=model.sample_rate)

MLX binds its Metal stream to the thread that loaded the model. Every request failed with RuntimeError: There is no Stream(gpu, 0) in current thread when the main thread loaded the model and an HTTP handler thread ran generation. A direct test settled the cause: generation from a worker thread succeeds when that same thread also performed the load, and fails when it did not.

I moved model loading and generation onto one dedicated engine thread, which owns the model for the lifetime of the process. HTTP handlers stay on their own threads. Each submits work to a queue and blocks until the result returns. The queue serializes generation, which fits a non-reentrant model when one generation already saturates the GPU.

Sheet 3 shows the weights resident in 1.1 seconds at startup. They never reload, so every request pays only for generation.

I also checked whether the pipeline watermarks its output. The mlx-audio package contains watermark encoder blocks that would embed an inaudible marker in generated audio^[8]. Those blocks live in the dacvae codec, while the Fish path loads fish_s1_dac. The Fish path never invokes the watermark module, so nothing is stamped into these files.

The app centers on a text box and voice dropdown. The Generate button sits beside the delivery tags, and each clip shows its metrics underneath.

About thirty inline tags control delivery at the word level: [pause], [whisper], [laughing], [sigh], [shouting], [excited], [with strong accent], and others. I insert them in the text where I want them to take effect.

Well, this is convenient. [pause] I can say anything now, [chuckle] and it never leaves this laptop.

A single generation can render multi-speaker dialogue through speaker tags. One pass keeps the acoustic conditions consistent between the two voices:

<|speaker:0|>So did it work? <|speaker:1|>[laughing] Yeah, completely offline.

Sampling controls sit on the same panel. Temperature and top-p at 0.7 give clean narration. A temperature near 1.0 adds variation in delivery and widens the range of speakers the model invents when unpinned. The preset factory in Sheet 5 uses that setting.

I'd add more voice diversity first. Seven harvested presets clustered between 130 and 202 Hz because that is where the model's unpinned sampling lives. More rolls might find distinct voices at the extremes. Real reference clips work far better than harvesting and are the model's intended path.

Pitch is a thin proxy for identity. Two voices at 172 Hz can sound nothing alike because a single number omits timbre and accent; it says nothing about pace either. I use it only to sort the dropdown. A speaker-embedding distance would order the library far better and let the factory reject a new roll for sounding similar even when its pitch differs.

A product using S2 Pro needs a separate commercial agreement. Without one, I would start with Chatterbox^[9], which is MIT and does voice cloning. Kokoro^[10] is Apache-2.0 and much smaller, though it has no cloning. Both run under the same MLX runtime, so switching requires a different model identifier.

Voice cloning also creates a consent problem. A reference clip works with anybody's voice, including people who never agreed to its use. The technical capability arrived well before any consensus about acceptable use, and a growing number of jurisdictions regulate it independently of the model license.

Speech is now a local function for me. I no longer budget for each draft-listen-revise pass, so I run it twenty times instead of three and get better writing from the extra work. A better, probably smaller model will replace these weights within a year. The engine thread and preset tooling, including calibration, should survive the swap because they do not depend on these weights.