I made this, so treat it as biased. It is free and open, with no paid tier or API behind it. Silencio-2 is a fine-tune of Kyutai's Pocket TTS 18M that follows tags written inside the text: What a long day. [sigh] Keep your voice down. Keep your voice down. I said stop! Weights and code: https://huggingface.co/BlueRoseLabs/Silencio-2 Try it in your browser, no setup: https://colab.research.google.com/drive/1d6GSiRgXbYyBo91DCMrSr5y2LR_peJKy?usp=sharing The numbers 18.2M parameters for speech, 27.9M with the encoder that voice cloning needs, 112 MB on disk About 20x realtime on an Apple M4 with 4 CPU threads, no GPU. That is one long-text run on my laptop, not a benchmark 24 kHz, streaming, voice cloning from a short clip, English only Sounds: laugh, sigh, gasp, cough, clear_throat Emotions: happy, sad, angry, fear, surprised, whisper Trained on 2x T4: about 50 hours of audio, 27k clips, roughly 6 hours for the final run Weights are CC-BY-4.0, inherited from Pocket TTS. The core inference code is MIT What went wrong, which is the useful part Run 1: tags trained fine and did almost nothing. Emotions changed the voice, but not into the emotion they named. Run 2: I generated better data and filtered it with AST, an AudioSet audio tagger, so only clips where the sound was detected would survive. The output looked clean. Then I counted rows by tag: zero laugh, zero cough, zero clear_throat. The filter had silently removed them, and the model never saw them. The sounds that survived were the only ones the detector could hear in the model's output. Run 3: I blamed the whole-clip scoring and switched to 3 second sliding windows. Human recordings of scripted laughter still scored 0 out of 23 above my threshold, with a median of 0.006. Whisper clips did not fare better. Then the run crashed anyway: 32 clips of 25 seconds in one batch do not fit on a T4. Run 4: I listened to the data, it was fine, and I removed the filter entirely. That is this model. My takeaway: a clip-level AudioSet tagger is a poor gate for short sounds mixed into speech. Count your rows per tag before you train, and spot check by ear. Two smaller fixes If the voice prompt is the start of an emotional clip, the model can copy the emotion from the prompt instead of following the tag. So half of the tagged samples take their prompt from a neutral take of the same voice. Pocket TTS splits long text into chunks, so a leading tag only covers the first chunk. The inference package repeats it on each chunk. What is still weak is the least stable tag, and it depends on the line and the voice. Other tags also vary from seed to seed, so try a few takes. A real whisper comes out about 10x quieter than speech, so generate() raises its level by default. I did not run a WER benchmark on this exact checkpoint. A later training stage that I did not ship scored 1.9% WER against 1.5% for base Pocket TTS on 300 LibriSpeech items, so expect a small cost in accuracy. There is no pip package yet: you clone the repo and install requirements.txt. Huge credit to Kyutai for Pocket TTS and for releasing the training code. None of this exists without it. Which tags would you actually use in a real project?   submitted by   /u/N1kuson [link]   [comments]