Use Google Cloud Speech in Twilio IVR - node.js

Question

I'm using Twilio's example of Google Cloud Speech to transcribe the audio stream of Twilio in real time (because twilio model doesn't support my language). But I wanted to response to user after I heard some keyword from them. But the gather function only works with Twilio's speech model.
app.post("/", (req, res) => {
res.set("Content-Type", "text/xml");
res.send(`
<Response>
<Start>
<Stream url="wss://${req.headers.host}/"/>
</Start>
<Say>Hello?</Say>
<Gather/>
</Response>
`);
});
I would like to know how to response to user after I catch the keyword, I tried to response in the stream but it doesn't work:
recognizeStream = client
.streamingRecognize(request)
.on("error", console.error)
.on("data", (data) => {
console.log("result: " + data.results[0].alternatives[0].transcript);
if (data.results[0].alternatives[0].transcript.toLowerCase() == "xin chào"){
console.log("got xin chao")
app.post("/", (req, res) => {
res.set("Content-Type", "text/xml");
res.send(`
<Response>
<Say>Hello there</Say>
</Response>
`);
});
}
Please help me out, thanks!

Answer 1

Twilio developer evangelist here.
When you use <Start><Stream> Twilio forks the audio and starts sending the audio from the call to your websocket. The call then carries on with the next TwiML instruction. If you want to interrupt the user once they say a keyword there are two approaches you can take.
Instead of using <Start><Stream> you can use <Connect><Stream>. <Connect> sets up a bidirectional stream, allowing you to receive the audio over the web socket connection and send audio back over the web socket. This way you could use the Google Cloud Speech API to generate your responses and stream them into the call. <Connect> doesn't fork the stream, so it won't carry out the TwiML instructions after it until the connection is ended.
Alternatively, you can continue using <Start><Stream> but once you get the keyword you are looking for you can redirect the call to new TwiML. Note that you need to use the REST API to update the call rather than returning TwiML within the web socket stream, like you are attempting.