Skip to content

zipvoice.md

Latest commit

 

History

History
191 lines (156 loc) · 10.5 KB

zipvoice.md

File metadata and controls

191 lines (156 loc) · 10.5 KB
Raw

ZipVoice (community model)

ZipVoice (k2-fsa) is a zero-shot voice-cloning TTS built on a TTSZipformer flow-matching backbone with a Vocos mel-24kHz vocoder. zipvoice_distill is the distilled variant (8 Euler steps, guidance-scale embedding); the base model uses batched classifier-free guidance. Both share one architecture and one GGUF packaging.

Status: inference complete. Text encoder, duration/ratio conditioning, flow-matching decoder, and the Vocos vocoder are verified against the reference PyTorch implementation with golden fixtures (tests/zipvoice/zipvoice_parity_main.cpp): per-layer and per-submodule encoder taps, text conditioning, sampled features, single- and batched velocity fields (odd/even/full lengths, both timesteps), fbank, and vocos audio — all at cosine 1.0. The Chinese/English text frontend (emilia mode) is verified token-for-token against the upstream EmiliaTokenizer (tests/zipvoice/zipvoice_zh_tokens_main.cpp).

Highlights

  • Zero-shot voice cloning from a short reference clip + transcript (zh and en)
  • Duration prediction from the prompt speaking rate (speed option scales it)
  • Distill model: 8 Euler steps with guidance-scale embedding; base model: batched CFG
  • Chinese + English mixed text via the emilia frontend (jieba + pypinyin tables + espeak-ng), inline pinyin overrides (<zhong1> <guo2>)
  • Ready-made GGUF package hosted at davidxifeng/zipvoice-gguf (Apache-2.0 upstream); local re-conversion with the tools below also works

Conversion

A ready-made, self-contained package is hosted at davidxifeng/zipvoice-gguf — the model manager downloads it directly (zipvoice-distill-orig.gguf, flow-matching model + bundled Vocos + embedded frontend sidecars). The Q8_0 package includes the same resources:

python3 tools/model_manager_v2.py install zipvoice_distill_q8_0

To rebuild it locally:

# 1. stage the Chinese frontend tables (requires the upstream ZipVoice python env
#    for pypinyin; writes zh_chars.tsv / zh_phrases.tsv / zh_syllables.tsv and
#    downloads the pinned jieba dictionaries next to tokens.txt)
python3 tools/community_models/export_zipvoice_zh_dict.py \
    --output-dir /models/ZipVoice/zipvoice_distill

# 2. flatten + package (torch checkpoint -> safetensors -> GGUF with model.* and vocos.*)
python3 tools/community_models/convert_zipvoice.py \
    --model-dir /models/ZipVoice/zipvoice_distill \
    --vocos /models/vocos-mel-24khz/vocos.safetensors \
    --converter build/bin/audiocpp_gguf

The converter needs tokens.txt + model.json + model.pt in --model-dir (the HF k2-fsa/ZipVoice zipvoice_distill snapshot layout). When the directory was staged by export_zipvoice_zh_dict.py, the zh_* frontend sidecars are embedded into the GGUF alongside tokens.txt / model.json, so a converted package is a single self-sufficient file (loose copies are still staged next to it for the directory layout). With --safetensors-only it stops at the development format (zipvoice-orig.safetensors + config + vocab in one directory) — an intermediate for tooling and the direct synthesis API; the CLI loads GGUF packages.

CLI usage

# English cloning (espeak frontend, the default)
audiocpp_cli --task clon --family zipvoice \
    --model /models/ZipVoice-Distill-GGUF/zipvoice-distill-orig.gguf \
    --session-option zipvoice.espeak_library_path=/opt/homebrew/lib/libespeak-ng.dylib \
    --voice-ref prompt.wav --reference-text "Reference transcript." \
    --text "Text to synthesize." --out out.wav

# Chinese / mixed text (emilia frontend, built in; the zh_* tables are embedded in the
# GGUF, or staged by export_zipvoice_zh_dict.py next to it)
audiocpp_cli --task clon --family zipvoice \
    --model ... \
    --session-option zipvoice.espeak_library_path=/opt/homebrew/lib/libespeak-ng.dylib \
    --voice-ref prompt.wav --reference-text "参考文本。" \
    --text "要合成的文本。" --out out.wav

Chunk streaming uses the same text-splitting path as offline long-form synthesis. Each chunk_<index> event is a complete ZipVoice synthesis for one text chunk; finish_stream() returns the concatenated 24 kHz waveform. It reduces time-to-first-event for long text, but it is not frame-level or token-level streaming.

audiocpp_cli --task clon --family zipvoice --mode streaming \
    --model /models/ZipVoice-Distill-GGUF/zipvoice-distill-orig.gguf \
    --voice-ref prompt.wav --reference-text "Reference transcript." \
    --text "A longer request is split into text chunks and emitted as named audio events." \
    --text-chunk-size 64 --out streamed.wav

GGUF packages created before streaming support may embed the older offline contract. Regenerate those packages with the current model_specs/zipvoice.json, or pass --model-spec-override model_specs/zipvoice.json when inspecting/loading them.

The text frontend is fixed to the EmiliaTokenizer pipeline (zh/en/mixed; the upstream default): tokenizer is no longer a session option. Session options: zipvoice.vocos_path (only for safetensors checkpoints without a bundled vocoder), zipvoice.espeak_library_path, zipvoice.espeak_data_path, zipvoice.num_inference_steps, zipvoice.guidance_scale, zipvoice.shift (legacy zipvoice.t_shift remains accepted). Requests accept reference_text (required), guidance_scale, num_inference_steps, shift (legacy t_shift), speed, feat_scale, target_rms, seed, language (legacy lang; espeak voice for English segments), text_chunk_size, and text_chunk_mode. Only the direct synthesis API accepts token_ids/prompt_token_ids to bypass the frontend; these are not CLI/server request options. Direct API callers can also select the espeak/simple frontends through ZipVoiceSynthesisRequest::tokenizer.

The audio.cpp session defaults are guidance_scale=3 and num_inference_steps=8 for both ZipVoice and ZipVoice-Distill. Request values override the session defaults.

Frontend details (emilia mode)

The upstream default EmiliaTokenizer pipeline is reproduced exactly: Chinese text normalization (framework ChineseTextNormalizer) → punctuation mapping → jieba segmentation (a model-local port of the Jieba maximum-probability DAG and four-state BMES HMM, src/community_models/zipvoice/jieba_segmenter.cpp, same dictionaries as python jieba) → pypinyin readings from baked tables (TONE3 syllables split into initial+0 / final+tone tokens) → tone sandhi (3rd-tone runs, 一/不) applied per jieba word → token mapping with OOV skipping; English runs are phonemized with espeak-ng. Verification compares C++ token ids against the upstream tokenizer run in the ZipVoice python environment (tests/zipvoice/zh_reference_ids.py).

The jieba segmentation port is derived from cppjieba (Copyright (c) 2013 Yanyi Wu) and jieba (Copyright (c) 2012 Sun Junyi), both MIT licensed; the zh_jieba_dict.txt / zh_hmm_model.txt resources are the jieba dictionaries and are embedded in the GGUF.

Performance

The measured ZipVoice-Distill ggml Metal runs achieve RTF < 1, the target for community models. RTF is synthesis wall time divided by generated audio duration; lower is better.

Measured on Apple M3, Metal backend, on 2026-09-18, with a Release build and 8 CPU threads, using zipvoice-distill-orig.gguf, 8 Euler steps, guidance scale 3, time shift 0.5, speed 1, seed 666, and the 24 kHz Vocos vocoder. Each reference pair was tested with one discarded warmup followed by three timed calls in the same process. Wall time includes tokenization, reference preprocessing, feature extraction, text encoding, the solver, and Vocos. Model loading and initial compilation are excluded by warmup; reference-file reading and output WAV writing are outside the timer.

Reference audio / transcript Generated audio Mean synthesis wall time Mean RTF Meets RTF < 1
zh-male-ref.wav / zh-male-ref.txt 19.115 s 4.697 s 0.2457 Yes
zh-ref.wav / zh-ref-text.txt 12.843 s 2.332 s 0.1816 Yes

The individual measured RTFs were 0.245406, 0.245910, 0.245799 for the male reference and 0.181118, 0.181592, 0.182100 for the female reference. Both used this exact target text, including the space in “宁 静”:

风声渐息，落叶不再沙沙作响。小狐狸缓缓合上双眼，在平缓绵长的呼吸声中，安然步入宁 静的梦境。

The measured implementation includes the session-owned runtime, framework Vocos graph, and model-local Jieba implementation. It measures the direct synthesis API with one persistent ZipVoiceComputeDevice per reference pair; session-level long-text chunking is not exercised. The two benchmarks ran sequentially. The discarded first calls took 6.744 s (male) and 2.772 s (female), including lazy model loading and graph/kernel setup but excluding resource-bundle resolution and input-file reading.

Both cases meet the community-model target. Results cover these two inputs on one machine; CPU and CUDA performance are not established by this measurement. Local benchmark source, full logs, and output WAVs are saved in outputs/zipvoice-metal-current-20260918/ (not distributed with the model).

Parity

Select --backend metal for Apple GPU inference or --backend cpu for the CPU reference path. The text encoder, flow decoder and Vocos backbone run on the selected backend; feature extraction and the final ISTFT run on the host. The direct C++ API defaults to BestAvailable, while sessions honor the requested backend, including an explicit CPU selection.

CPU/Metal parity is tested on Apple M3 with the Distill GGUF, including odd sequence lengths, batched velocity evaluation, Vocos and eight-step synthesis. ZipVoice requests F32 matrix products to avoid FP16 staging error accumulating through the sampler. CUDA uses the same graphs but has not been verified on hardware in this GPU validation run.

# golden fixtures: tests/zipvoice/build_reference.py (upstream env) + parity harness
build/bin/zipvoice_parity reference.npz <model-path> reference.npz vocos.safetensors out.wav
build/bin/zipvoice_zh_tokens zh_reference_ids.json <model-dir>

Weights and runtime graphs are owned by the ZipVoiceComputeDevice runtime (one per session), so callers injecting a borrowed backend must release the device runtime (for example with zipvoice_clear_runtime(device)) before destroying that backend, after all ZipVoice calls using it have finished.