chtbx-livive-net

Chatterbox voice-clone experiment

Zero-shot voice cloning with Chatterbox: clone a voice from a short reference clip and synthesize arbitrary text.

Target phrase for this experiment: “Artificial Intelligence & Machine Learning”. Rendered for the portfolio audio at https://chtbx.livive.net/audio/v1/.

Repeatable workflow

# 1. One-time setup: create .venv (Python 3.14) and install chatterbox-tts
./install.sh

# 2. Record ~10s of your voice -> ref.wav
./record_ref.sh ref.wav

#    Or convert an existing recording (QuickTime, Voice Memos, iPhone):
./prep_ref.sh ~/Desktop/my-voice.m4a ref.wav

# 3. Synthesize the phrase in your cloned voice
.venv/bin/python gen.py \
  --ref ref.wav \
  --text "Artificial Intelligence & Machine Learning" \
  --out out/voice.wav

# 4. Listen
afplay out/voice.wav

Or via make: make setup && make record && make say && make preview.

Web audio for chtbx.livive.net

Render a whole set of phrases (category titles, taglines) as web-optimized audio and ship them from this repo (CNAME chtbx.livive.net, remote jpeckenpaugh/chtbx-livive-net):

make web                     # renders phrases.txt -> audio/v1
# or explicitly:
.venv/bin/python render_phrases.py --ref refs/jarad.m4a --out-dir audio/v1

The model loads once for the whole batch. Existing clips are skipped unless you pass --force. Edited phrases.txt? Just re-run.

Why audio/v1/ and not build/, dist/, or cdn/

GitHub Pages serves the repo root as-is; there is no build step, so build/ and dist/ (which conventionally mean generated, gitignored output) would be misleading, and cdn/ would imply a separate origin. audio/ matches the repo’s root-level topic folders. The v1/ segment versions the set so a re-render can be published atomically under a new path while the old one stays cached — bump it when the phrase set changes materially.

Format recommendation: MP3, mono, 24 kHz, 64 kbps

Option Verdict
MP3 (chosen) Universally decodable; GitHub Pages serves .mp3 as audio/mpeg.
AAC / .m4a Good for Apple, but Firefox-on-Linux may lack an AAC decoder, and Pages may not map the MIME type.
Opus in .webm/.opus Best quality-per-byte (~half the size), but Safari/Opus container support is uneven and Pages likely serves .opus as application/octet-stream, which can break <audio>.

These are 1–3 second clips, so size is a non-issue (~224 KB for all 13). Prefer the format with zero playback risk. Keep MP3 mono at the source’s 24 kHz — no resampling, no stereo, and CBR is fine at these sizes.

<audio> MIME types matter: Pages serves known extensions (video/webm, image/webp) with correct Content-Type, but unknown ones fall back to application/octet-stream, which some browsers refuse to play.

Levels

Clips are normalized to a common speech level (active RMS, default -25 dBFS) with a -1 dBFS safety limiter. This is deliberately not EBU R128 loudnorm: these short clips have 18–24 dB crest factors, so loudnorm caps its gain at the true-peak ceiling and each clip lands wherever its own dynamics put it (measured −26…−20 LUFS when targeting −16 — a 6 dB spread). Speech-level normalization brought the set to roughly a 1 dB spread with no limiting engaged.

Naming and layout

Files are named by slugified phrase, e.g. It's a live vive → its-a-live-vive.mp3, under audio/v1/.

To use a clip:

<audio src="/audio/v1/its-a-live-vive.mp3" preload="none"></audio>

Reference clip guidelines

Tuning knobs

Passed through gen.py:

Flag Default Effect
--exaggeration 0.5 Expressiveness. Higher = more dramatic, slightly faster.
--cfg-weight 0.5 Pacing/guidance. Lower (~0.3) for slower, deliberate speech. Use 0 to drop the reference accent.
--temperature 0.8 Sampling randomness. Lower = more stable.
--seed none Fix for reproducible output.
--device auto cuda → mps → cpu. This Mac uses mps.

Reproducibility

Reading the debug output

gen.py silences the known-benign chatter from third-party libraries, so a clean run only shows real information:

Suppressed message Why it appears Handled by
pkg_resources is deprecated perth imports pkg_resources; that is why setuptools<81 is pinned warnings filter
``LoRACompatibleLinear is deprecated diffusers==0.29.0 constructs the class warnings filter
``torch.backends.cuda.sdp_kernel() is deprecated Chatterbox internals; a no-op on MPS warnings filter
sdpa` attention does not support `output_attentions=True transformers logger.warning_once logging filter

Note: installing peft does not silence the LoRACompatibleLinear warning. That warning is emitted inside LoRACompatibleLinear.__init__ on every construction — not an “is peft installed?” check — so only a diffusers upgrade would remove it. peft is therefore not a dependency here.

Remaining output worth acting on:

Message Meaning / fix
You are sending unauthenticated requests to the HF Hub Set HF_TOKEN (e.g. export HF_TOKEN=hf_...) for faster, rate-limit-free model downloads. Only matters on first run; the weights are cached afterwards
loaded PerthNet (Implicit) at step 250,000 Info — watermarker ready
Sampling: N% Progress bar (generation stops early at end-of-utterance)
PySoundFile failed. Trying audioread instead Should no longer occur: gen.py converts non-WAV references to 24 kHz mono WAV first. If you see it, ffmpeg was not on PATH, or you called prepare_conditionals directly
wrote ... (0.5s @ 24000 Hz) for a long sentence Early EOS / truncation. Retry with a different --seed, raise --cfg-weight, or record a cleaner reference

Layout

install.sh        # create .venv + install deps
record_ref.sh     # record a reference clip
prep_ref.sh       # convert an existing recording to ref.wav
check_level.py    # (used by the two above) duration/level verdict
gen.py            # clone + synthesize one phrase
render_phrases.py # batch render phrases.txt -> web audio
phrases.txt       # phrase manifest (category titles, taglines)
Makefile          # setup / record / prep / say / web / preview / clean
requirements.txt  # pinned chatterbox-tts
audio/v1/         # shipped web audio (committed)
out/              # generated audio + intermediate WAVs (gitignored)