CMU Sphinx and time indexed words - speech-to-text

Question

I have run the example of Pocketsphinx Python and now I am facing the issue that I want to run a 60sec wav file for speech recognition in English and want as output
- the English translation AND
- at which second each word was mentioned.
Now, I do not know where to start to dome some research to get the required output. Could anyone please point me in the right direction??

Answer 1

ok, the open source tools like Kaldi automatically offers this:
https://americanarchivepb.wordpress.com/2017/12/04/dockerized-kaldi-speech-to-text-tool/

Answer 2

You need recognition with forced alignment. Here is an example for pocketsphinx:
pocketsphinx_continuous
-infile with.wav
-jsgf with-word.jsgf
-dict words.dict
-backtrace yes
-fsgusefiller no
-bestpath no
2>&1 > with-word.txt
Output:
==> with-word.txt <==
INFO: fsg_search.c(869): fsg 0.05 CPU 0.051 xRT
INFO: fsg_search.c(871): fsg 0.09 wall 0.084 xRT
INFO: pocketsphinx.c(1171): sil with sil (-2607)
word start end pprob ascr lscr lback
sil 3 77 1.000 -1602 0 1
with 78 102 1.000 -845 0 1
sil 103 107 1.000 -160 0 1
INFO: fsg_search.c(265): TOTAL fsg 0.05 CPU 0.051 xRT
INFO: fsg_search.c(268): TOTAL fsg 0.09 wall 0.085 xRT
sil with sil
For CMU Sphinx 4 you need the SpeechAligner class from Sphinx API. Here you'll find an implementation of simple aligner tool.
./align.sh sample.wav sample.txt 2>/dev/null
Output:
"it's","IH T S","false","0.0","170","200"
"a","AH","false","-5540774.0","200","390"
"crowd","K R AW D","false","-1.13934288E8","850","1300"
"in","IH N","false","-1.95127088E8","1300","1470"
"two","T UW","false","-2.23176048E8","1470","1700"
"distinct","D IH S T IH NG K T","false","-2.6345264E8","1700","2230"
"ways","W EY Z","false","-3.58427808E8","2230","2730"
"the","DH AH","false","-4.72551168E8","2920","3100"
"fruit","F R UW T","false","-5.24233504E8","3220","3530"
"of","AH V","false","-5.79971456E8","3530","3640"
"a","AH","false","-5.99515456E8","3640","3760"
"figg","F IH G","false","-6.2017152E8","3760","4060"
"tree","T R IY","false","-6.72126656E8","4060","4490"
"is","IH Z","false","-7.4763744E8","4490","4570"
"apple","AE P AH L","false","-7.73581184E8","4630","5040"
"shaped","SH EY P T","false","-8.44424704E8","5040","5340"