I'd like to create a custom voice in Mozilla TTS using audio samples I have recorded but am not sure how to get started. The Mozilla TTS project has documentation and tutorials, but I'm having trouble putting the pieces together -- it seems like there's some basic information missing that someone starting out needs to know to get going.
Some questions I have:
I see that there is a Docker image for Mozilla TTS, but that the documentation for it covers creating speech and doesn't mention training. Can I use the Docker image for training?
If I can't use the Docker image for training, how do I get a functional copy of Mozilla TTS running on my system with Python 3? I've tried following the commands that the project provides, but I get dependency errors, version conflicts, or errors about not having sufficient permission to install packages.
What information do I need in order to train the model? What audio formats do I need? I see that I need a metadata.csv file -- what do I need to put in that file? What do I customize in the config file?
Most of the configs reference a scale_stats.npy file -- how do I generate this?
How do I run the training?
After a lot of research and experimentation, I can share my learnings to answer my own questions.
Can the Mozilla TTS Docker image be used for training (TL;DR: "No")
The Mozilla TTS docker image is really geared for playback and doesn't seem equipped to be used for training. At least, even when running a shell inside the container, I could not get training to work. But after figuring out what was causing PIP to be unhappy, the process of getting Mozilla TTS up and running in Ubuntu turns out to be pretty straightforward.
Installing Mozilla TTS using Python 3, PIP, and a Virtual Environment
The documentation for Mozilla TTS doesn't mention anything about virtual environments, but IMHO it really should. Virtual environments ensure that dependencies for different Python-based applications on your machine don't conflict.
I'm running Ubuntu 20.04 on WSL, so Python 3 is already installed. Given that, from within my home folder, here are the commands I used to get a working copy of Mozilla TTS:
sudo apt-get install espeak
git clone https://github.com/mozilla/TTS mozilla-tts
python3 -m venv mozilla-tts
cd mozilla-tts
./bin/pip install -e .
This created a folder called ~/mozilla-tts in my home folder that contains the Mozilla TTS code. The folder is setup as a virtual environment, which means that as long as I execute python commands via ~/mozilla-tts/bin/python and PIP via ~/mozilla-tts/bin/pip, Python will use only the packages that exist in that virtual environment. That eliminates the need to be root when running pip (since we're not affecting system-wide packages), and it ensures no package conflicts. Score!
Prerequisites for Training a Model
For the best results when training a model, you will need:
Short audio recordings (at least 100) that are:
In 16-bit, mono PCM WAV format.
Between 1 and 10 seconds each.
Have a sample rate of 22050 Hz.
Have a minimum of background noise and distortion.
Have no long pauses of silence at the beginning, throughout the middle, and at the end.
A metadata.csv file that references each WAV file and indicates what text is spoken in the WAV file.
A configuration file tailored to your data set and chosen vocoder (e.g. Tacotron, WavGrad, etc).
A machine with a fast CPU (ideally an nVidia GPU with CUDA support and at least 12 GB of GPU RAM; you cannot effectively use CUDA if you have less than 8 GB OF GPU RAM).
Lots of RAM (at least 16 GB of RAM is preferable).
Preparing the Audio Files
If your source of audio is in a different format than WAV, you will need to use a program like Audacity or SoX to convert the files into WAV format. You should also trim out portions of audio that are just noise, umms, ahs, and other sounds from the speaker that aren't really words you're training on.
If your source of audio isn't perfect (i.e. has some background noise), is in a different format, or happens to be a higher sample rate or different resolution (e.g. 24-bit, 32-bit, etc.), you can perform some clean-up and conversion. Here's a script that is based on an earlier script from the Mozilla TTS Discourse forums:
from pathlib import Path
import os
import subprocess
import soundfile as sf
import pyloudnorm as pyln
import sys
src = sys.argv[1]
rnn = "/PATH/TO/rnnoise_demo"
paths = Path(src).glob("**/*.wav")
for filepath in paths:
target_filepath=Path(str(filepath).replace("original", "converted"))
target_dir=os.path.dirname(target_filepath)
if (str(filepath) == str(target_filepath)):
raise ValueError("Source and target path are identical: " + str(target_filepath))
print("From: " + str(filepath))
print("To: " + str(target_filepath))
# Stereo to Mono; upsample to 48000Hz
subprocess.run(["sox", filepath, "48k.wav", "remix", "-", "rate", "48000"])
subprocess.run(["sox", "48k.wav", "-c", "1", "-r", "48000", "-b", "16", "-e", "signed-integer", "-t", "raw", "temp.raw"]) # convert wav to raw
subprocess.run([rnn, "temp.raw", "rnn.raw"]) # apply rnnoise
subprocess.run(["sox", "-r", "48k", "-b", "16", "-e", "signed-integer", "rnn.raw", "-t", "wav", "rnn.wav"]) # convert raw back to wav
subprocess.run(["mkdir", "-p", str(target_dir)])
subprocess.run(["sox", "rnn.wav", str(target_filepath), "remix", "-", "highpass", "100", "lowpass", "7000", "rate", "22050"]) # apply high/low pass filter and change sr to 22050Hz
data, rate = sf.read(target_filepath)
# peak normalize audio to -1 dB
peak_normalized_audio = pyln.normalize.peak(data, -1.0)
# measure the loudness first
meter = pyln.Meter(rate) # create BS.1770 meter
loudness = meter.integrated_loudness(data)
# loudness normalize audio to -25 dB LUFS
loudness_normalized_audio = pyln.normalize.loudness(data, loudness, -25.0)
sf.write(target_filepath, data=loudness_normalized_audio, samplerate=22050)
print("")
To use the script above, you will need to check out and build the RNNoise project:
sudo apt update
sudo apt-get install build-essential autoconf automake gdb git libffi-dev zlib1g-dev libssl-dev
git clone https://github.com/xiph/rnnoise.git
cd rnnoise
./autogen.sh
./configure
make
You will also need SoX installed:
sudo apt install sox
And, you will need to install pyloudnorm via ./bin/pip.
Then, just customize the script so that rnn points to the path of the rnnoise_demo command (after building RNNoise, you can find it in the examples folder). Then, run the script, passing the source path -- the folder where you have your WAV files -- as the first command line argument. Make sure that the word "original" appears somewhere in the path. The script will automatically place the converted files in a corresponding path, with original changed to converted; for example, if your source path is /path/to/files/original, the script will automatically place the converted results in /path/to/files/converted.
Preparing the Metadata
Mozilla TTS supports several different data loaders, but one of the most common is LJSpeech. To use it, we can organize our data set to follow LJSpeech conventions.
First, organize your files so that you have a structure like this:
- metadata.csv
- wavs/
- audio1.wav
- audio2.wav
...
- last_audio.wav
The naming of the audio files doesn't appear to be significant. But, the files must be in a folder called wavs. You can use sub-folders inside wavs though, if so desired.
The metadata.csv file should be in the following format:
audio1|line that's spoken in the first file
audio2|line that's spoken in the second file
last_audio|line that's spoken in the last file
Note that:
There is no header line.
The columns are joined together with a pipe symbol (|).
There should be one row per WAV file.
The WAV filename is in the first column, without the wavs/ folder prefix, and without the .wav suffix.
The textual description of what's spoken in the WAV is written out in the second column, with all numbers and abbreviations spelled-out.
(I did observe that steps in the documentation for Mozilla TTS have you then shuffle the metadata file and then split it into a "training" set (metadata_train.csv) and "validation" set (metadata_val.csv), but none of the sample configs provided in the repo are actually configured to use these files. I've filed an issue about that because it's confusing/counter-intuitive to a beginner.)
Preparing the config.json File
You need to prepare a configuration file that describes how your custom TTS will be configured. This file is used by multiple parts of Mozilla TTS when preparing for training, performing training, and generating audio from your custom TTS. Unfortunately, though this file is very important, the documentation for Mozilla TTS largely glosses over how to customize this file.
To start, create a copy of the default Tacotron config.json file from the Mozilla repo. Then, be sure to customize at least the audio.stats_path, output_path, phoneme_cache_path, and datasets.path file.
You can customize other parameters if you so choose, but the defaults are a good place to start. For example, you can change the run_name to control the naming of folders containing your datasets.
Do not change the datasets.name parameter (leave it set to "ljspeech"); otherwise you'll get strange errors related to an undefined dataset type. It appears that the dataset name refers to the type of data loader used, rather than what you call your data set. Similarly, I haven't risked changing the model setting, since I don't yet know how that value gets used by the system.
Preparing scale_stats.npy
Most of the training configurations rely on a statistics file called scale_stats.npy that's generated based on the training set. You can use the ./TTS/bin/compute_statistics.py script inside the Mozilla TTS repo to generate this file. This script requires your config.json file as an input, and is a good step to sanity check that everything looks good up to this point.
Here's an example of a command you can run if you are inside the Mozilla TTS folder you created at the start of this tutorial (adjust paths to fit your project):
./bin/python ./TTS/bin/compute_statistics.py --config_path /path/to/your/project/config.json --out_path /path/to/your/project/scale_stats.npy
If successful, this will generate a scale_stats.npy file under /path/to/your/project/scale_stats.npy. Be sure that the path in the audio.stats_path setting of your config.json file matches this path.
Training the Model
It's now time for the moment of truth -- it's time to start training your model!
Here's an example of a command you can run to train a Tacotron model if you are inside the Mozilla TTS folder you created at the start of this tutorial (adjust paths to fit your project):
./bin/python ./TTS/bin/train_tacotron.py --config_path /path/to/your/project/config.json
This process will take several hours, if not days. If your machine supports CUDA and has it properly configured, the process will run more quickly than if you are just relying on CPU alone.
If you get any errors related to a "signal error" or "signal received", this typically indicates that your machine does not have enough memory for the operation. You can run the training with less parallelism but it will run much more slowly.
Note, on windows, following GuyPaddock's advice from prior, I had to use pip install -e. instead of leading with ./bin/pip, and I had to use python instead of python3
Might be obvious to someone else but I am not so familiar with python or path shortcuts in shell being customized etc.
I am trying to use EECodes in python to get various weather information, such as temperature, humidity, etc out of grib2 files. I am using the GFS files. I would like to be able to extract the data as (lat,lon,alt,$data_point), and as a 2d array for each altitude.
I have tried the example programs located here: https://confluence.ecmwf.int/display/ECC/grib_iterator_bitmap
I can't figure out what I am looking in the output of that program. When I load the messages using their keys, it is not obvious how to make a grid. When I load the grid, the data doesn't have labels I understand.
#craeft have a look to https://github.com/ecmwf/cfgrib. cfgrib is the new standard for python and grib file handling. It is easy to install and easy to access files. Please install the latest version because it supports GFS files.
I have a client that created several large PDFs, each containing hundreds of images within them. The images were created with a program that adds unique info to each file; random binary data was placed in some file headers, some files have data disguised as image artifacts, and general metadata in each image. While I'm unfamiliar with the program, I understand that it's a marketing software suite of some sort so I assume the data is used for tracking online distribution and analytics.
I have the source files used to create the PDFs and while I could open each image, clone its visual data, strip metadata and re-compress the images to remove the identification data, I would much rather automate the process using Pillow. The problem is, I'm worried I could miss something. The client is hoping to release the files from behind an online username, and he doesn't want the username tied to this program or its analytical tracking mechanisms.
So my question is this: how would I clone an image with Pillow in a way that would strip all identifying metadata? The image files are massive, ranging from 128MB to 2GB. All of the images are either PNG uncompressed or JPEG files with very mild compression. I'm not married to Pillow, so if there's a better software library (or standalone software) that better suits this, I'll use it instead.
Just use ImageMagick as installed on most Linux distros and available for macOS and Windows. So, in Terminal, strip the metadata from an input file and resave:
magick input.jpg -strip result.jpg
If you want to do all JPEGs in current directory:
magick mogrify -strip *.jpg
Or maybe you want to change the quality a little too:
magick mogrify -quality 82 -strip *.jpg
Copying the pixel data to a new image should remove all metadata and compressing the image slightly as a jpeg should remove steganographic tracking data.
You may have to modify the load/copy/save methods to deal with large files. Also pay attention to the PIL file size limitations. Opacity in png files isn't handled here.
import os
from PIL import Image
picture_dir = ''
for subdir, dirs, files in os.walk(picture_dir):
for f in files:
ext = os.path.splitext(f)[1]
if( ext in ['.jpg','.jpeg','.png'] ):
full_path = os.path.join(subdir, f)
im = Image.open(full_path)
data = list(im.getdata())
no_exif = Image.new('RGB', im.size) # not handling opacity
no_exif.putdata(data) # should strip exif
out_path = full_path.split(ext)[0] + 'clean.jpg'
no_exif.save(out_path, 'JPEG', quality=95) # compressing should remove steganography
I have 2 functions as below:
def select_audio():
os.chdir("/home/norman/songbook")
top1.lower(root)
name=tkFileDialog.askopenfilename()
doit="play " + name
top1.lift(root)
os.system(doit)
def select_video():
os.chdir("/home/norman/Videos")
top2.lower(root)
name=tkFileDialog.askopenfilename()
doit="mpv --fs " + name
top2.lift(root)
os.system(doit)
They are selected from buttons to allow choosing and playing audio files or video files.
They work to some extent.
Videos are in a different directory and at the same level as the audio files.
It doesn't matter which I choose first I see the correct directory so I can play say a video, if after it's finished I choose audio it still shows the video directory.
Similarly if I first choose audio it still shows the audio directory if I select videos.
I have no idea why it does this. I am not an experienced programmer as you can probably tell from the code.
Some suggestions:
Use a raw string to make sure that Python doesn't try to interpret anything following a \ as an escape sequence:
Change os.chdir("/home/norman/whatever") to os.chdir(r"/home/norman/whatever")
It won't solve this problem, but it will avoid you future problems.
For tkFileDialog use the initialdir option:
Change name=tkFileDialog.askopenfilename() to
name=tkFileDialog.askopenfilename(initialdir=r"home/norman/whatever", parent=root)
I'm looking to combine a range of different audio files (mp3) in Python. One of the requirements is that I need to be able to specify a delay at the end of each file. To illustrate, something like:
[file1.mp3--------3 seconds----------][delay---------2 seconds--------][file2.mp3]-------------4 seconds][delay---------2 seconds][file3.mp3----------3 seconds---------]
Does anyone here know of any mp3 libraries that can accomplish this? Python isn't really a necessity here. If it'll be easier in another language, that'll be fine.
I think FFmpeg can do this, given the right arguments. No real need to use a library.
To combine wav or aiff files, you can do something like this: (inspiration from here)
import aifc
def concatenate(*items):
data = []
for item in items:
f = aifc.open(item, 'rb')
data.append([f.getparams(), f.readframes(f.getnframes())])
f.close()
output = aifc.open('output.aif', 'wb')
output.setparams(data[0][0])
for item in data:
output.writeframes(item[1])
output.close()
See the link for the wav format (it's pretty much the same, but with the wave library)
To add silence, I would just make a one second silent file using your favorite audio editor and then concatenate in the proper amount of silence.