Extract AUDIO, manipulate & merge again - linux

Question

I'm using Spleeter to remove music from audios.
My goal is to build a script that automates the process of extracting audio from the video, execute Spleeter on the extracted audio & than merge the manipulated audio back to the video replacing the original one.
The main issue I had is that I don't have enough ram to process the whole extracted audio. I need to split it the into multiple pieces & execute Spleeter upon each piece.
Then concatenate the manipulated pieces together and merge the result to the video.
Here's what I tried:
#!/bin/bash
cd ~/Desktop/Video-convert
# create audio from video
ffmpeg -i *.mp4 output.mp3
# Split the audio into pieces
ffmpeg -i output.mp3 -f segment -segment_time 120 -c copy output_%03d.mp3
# Execute Spleeter upon each sample
FILES=~/Desktop/Video-convert/*.mp3
for f in $FILES
do
spleeter separate -i $f -o output_vocal
done
# delete unneeded audios
rm *.mp3
cd output_vocal
# ===========================================================
# the problem starts here
# ===========================================================
# concatenate manipulated audios together
find . -name 'vocals.wav' -exec echo {} >> mylist.txt \;
ffmpeg -f concat -safe 0 -i mylist.txt -c copy vocal.mp3
mv vocal.mp3 ../
cd ../
# merge the audio back to video
ffmpeg -i *.mp4 -i vocal.mp3 \
-c:v copy -c:a aac -strict experimental \
-map 0:v:0 -map 1:a:0 vocal-vid.mp4
Everything works well until having to concatenate the audios together. Spleeter outputs the result into vocal.wav & accompaniment.wav within a sub-folder that is named the same as the audio that was processed.
The File Tree looks like this:
output_vocal
- output_000
----- vocal.wav
----- accompaniment.wav
- output_001
----- vocal.wav
----- accompaniment.wav
- output_002
----- vocal.wav
----- accompaniment.wav
As you can see the problem comes with the naming. My objective is to concatenate all vocal.wav into one mp3 audio.
And then merge the final vocal.mp3 audio with the *.mp4 video.
Only issue is going around the way that Spleeter outputs the result audios.

Answer 1

the problem you are experiencing is that ffmpeg's concat demuxer requires an input file that contains directives, rather than a naive file-list.
Your find invocation creates a file like:
output_vocal/output_000/vocal.wav
output_vocal/output_001/vocal.wav
output_vocal/output_002/vocal.wav
whereas ffmpeg's concat demuxer really requires a file like:
file output_vocal/output_000/vocal.wav
file output_vocal/output_001/vocal.wav
file output_vocal/output_002/vocal.wav
Also note that find does not necessarily return the files in alphabetic order, whereas you will most likely want to concatenate the files in that order.
Finally, when concatenating the WAV-files, you cannot use the copy codec to generate an MP3 file (since the WAV/RIFF codec is not MP3). but you don't need an intermediate MP3-file anyhow
Here's an updated script, that
- uses a temporary directory for all intermediate files
- iterates over all mp4-files provided at the cmdline (rather than hardcoding the input directory)
- creates a "XXX_voc.mp4" file for each input file "XXX.mp4" (overwriting any existing files)
#!/bin/bash
for infile in "$#"
do
outfile=${infile%.mp4}_voc.mp4
# create a temp-directory to put our stuff to
TMPDIR=$(mktemp -d)
# create audio from video
ffmpeg -i "${infile}" "${TMPDIR}/output.mp3"
# Split the audio into pieces
ffmpeg -i "${TMPDIR}/output.mp3" -f segment -segment_time 120 -c copy "${TMPDIR}/output_%03d.mp3"
# Execute Spleeter upon each sample
find "${TMPDIR}" -maxdepth 1 -type f -name "output_*.mp3" \
-exec spleeter separate -i {} -o "${TMPDIR}/output_vocal" ";"
# find all 'vocal.wav' files generated by spleeter, sort them,
# prefix them with 'file ', and put them into output.txt
find "${TMPDIR}/output_vocal" -type f -name "vocal.wav" -print0 \
| sort -z \
| xargs -0 -I{} echo "file '{}'" \
> "${TMPDIR}/output.txt"
# concatenate the files and create an MP3 file
ffmpeg -f concat -safe 0 -i "${TMPDIR}/output.txt" -c copy "${TMPDIR}/vocal.wav"
# merge the audio back to video
ffmpeg -y -i "${infile}" -i "${TMPDIR}/vocal.wav" \
-c:v copy -c:a aac -strict experimental \
-map 0:v:0 -map 1:a:0 "${outfile}"
rm -rf "${TMPDIR}"
done