Microsoft Azure Web Chat, trigger mic from other function - node.js

I have used Microsoft Bot Framework to create a bot for the client side. i.e. called WebChat. I have also added Speech SpeechRecognizer. However, I am trying to trigger a mic when a phrase is recited.
I couldn't find a function in Microsoft that does this. So I added my own speech recognizer that is called every second and once the phrase is called I want to call the mic function from the Cognitive Services.
How Can I achieve this?
I have got the speech recognizer from here
And the one I have written to identify a phrase is this:
function startDictation() {
if (window.hasOwnProperty('webkitSpeechRecognition')) {
var recognition = new webkitSpeechRecognition();
recognition.continuous = false;
recognition.interimResults = false;
recognition.lang = "en-US";
recognition.start();
recognition.onresult = function (e) {
var foundText = e.results[0][0].transcript;
console.log(foundText);
if (foundText == "hello hello") {
console.log("found text");
//call cognitive service mic function
recognition.stop();
}
else {
console.log("text not found");
recognition.stop();
startDictation();
}
};
recognition.onerror = function (e) {
console.log("found error", e);
recognition.stop();
}
}
}
Do let me know if any information is missing or miss-leading.
For more:

I tried to leverage startRecognizing() function in SpeechRecognizer class at https://github.com/Microsoft/BotFramework-WebChat/blob/master/src/CognitiveServices/SpeechRecognition.ts#L72 to trigger the recognize functionality. However, I found that only if I have click the mic item, then I could use startRecognizing() function to recognize the voice.
There is a trick workaround you can try to use at present:
I inspected the mic item, and try to triiger its click event in js, which worked exactly to recognize my speech.
You can try to use following js code snippet with jQuery:
$('.wc-mic').trigger("click")
Hope it helps.

Related

Unable to stream microphone audio to Google Speech to Text with NodeJS

I am going to develop a simple web based Speech to Text project. Develop with NodeJS, ws (WebSocket), and Google's Speech to Text API.
However, I have no luck to get the transcript from Google's Speech to Text API.
Below are my server side codes (server.js):
ws.on('message', function (message) {
if (typeof message === 'string') {
if(message == "connected") {
console.log(`Web browser connected postback.`);
}
}
else {
if (recognizeStream !== null) {
const buffer = new Int16Array(message, 0, Math.floor(message.byteLength / 2));
recognizeStream.write(buffer);
}
}
});
Below are my client side codes (ws.js):
function recorderProcess(e) {
var floatSamples = e.inputBuffer.getChannelData(0);
const ConversionFactor = 2 ** (16 - 1) - 1;
var floatSamples16 = Int16Array.from(floatSamples.map(n => n * ConversionFactor));
ws.send(floatSamples16);
}
function successCallback(stream) {
window.stream = stream;
var audioContext = window.AudioContext;
var context = new audioContext();
var audioInput = context.createMediaStreamSource(stream);
var recorder = context.createScriptProcessor(2048, 1, 1);
recorder.onaudioprocess = recorderProcess;
audioInput.connect(recorder);
recorder.connect(context.destination);
}
When I run the project, and open http://localhost/ in my browser, trying to speaking some sentences to the microphone. Unfortunately, there are no transcription returned, and no error messages returned in NodeJS console.
When I check the status in Google Cloud Console, it only display a 499 code in the dashboard.
Many thanks for helping!
I think the issue could be related to the stream process. Maybe some streaming process is stopped before the end of an operation. My suggestion is to review the callbacks in the JasvaScript code in order to find some “broken" promises.
Also, maybe its obvious but there is a different doc for audios than more than a minute:
https://cloud.google.com/speech-to-text/docs/async-recognize
CANCELLED - The operation was cancelled, typically by the caller.
HTTP Mapping: 499 Client Closed Request
Since the error message, this also could be related to the asynchronous and multithread features of node js.
Hope this works!

How to add Get Started button in the typing bar using bot builder sdk for node.js

I am using bot builder sdk for node.js to create a chatbot. Also connected it to facebook channel. I am using the following code to greet the user:
var bot = new builder.UniversalBot(connector, [
(session, result, next) => {
let text = '';
switch(session.message.address.channelId) {
case 'facebook':
text = 'Hi ' + session.message.user.name + ' !';
break;
default:
text = 'Hi !';
}
session.sendTyping();
session.say(text);
next();
},
(session, say) => {
}
]);
The above code works fine, but I want to add "Get Started" button in the typing bar to invoke the above code. Note that this button appears only once. Please find image of the typing bar below:
Is there a way to achieve this using bot builder sdk for node.js ?
Thanks
Although one can certainly add a button to start any activity with the bot, but that will limit the bots potential to only one customizable channel, i.e. WebChat.
I think there are better 2 alternative ways to get the desired functionality which will work across many channels.
First
I would suggest to add a conversation update event. Code goes in the botbuilder's middleware. Here is a sample code from the docs.
bot.on('conversationUpdate', function (message) {
if (message.membersAdded && message.membersAdded.length > 0) {
// Say hello
var txt = "Send me a Hi";
var reply = new builder.Message()
.address(message.address)
.text(txt);
bot.send(reply);
});
What this will do is make the bot send a message Send me a Hi to the user, if it determines this is a first time visitor. This will give the visitor enough cue to send the bot Hi by typing it. Although he can enter whatever he wants, but this will result in the invocation of the 1st dialog configured which in this case is the will be the dialog which you have posted in question.
Second
You can mark some dialog to be invoked automatically if your bot has never encountered this visitor. Here is the sample code...
var bot = new builder.UniversalBot(connector);
bot.dialog('firstRun', function (session) {
session.userData.firstRun = true;
session.send("Hello...").endDialog();
}).triggerAction({
onFindAction: function (context, callback) {
// Only trigger if we've never seen user before
if (!context.userData.firstRun) {
// Return a score of 1.1 to ensure the first run dialog wins
callback(null, 1.1);
} else {
callback(null, 0.0);
}
}
});
Here we have split the bot creation and dialog registration in 2 steps. And while registering the firstRun dialog, we have provided it the triggerAction that if the visitor is new, then trigger this dialog.
Both of these approaches do not use adding some extra buttons and it is up to the bot either to educate him on sending some message which in turn will start the 1st dialog or directly start some dialog.
For more info on conversationEvent you can refer to this page
I tried the above options, but they didn't seem to be working for facebook messenger. But I found a solution to add the Get Started button into the typing bar of the messenger. For that we need to use the Facebook Graph API and not the bot builder sdk.
https://graph.facebook.com/v2.6/me/messenger_profile?access_token=<PAGE_ACCESS_TOKEN>
{
"get_started":{
"payload":"Get Started"
}
}
The above API call will add the button for you to get the conversation started.
Thanks all for the help!!

Trigger a specific dialog in Bot via Directline API

I'm working on breaking my bot repo into 2 separate repos
A repo to purely handle bot logic
A repo to handle custom chat via directline
Currently , we have a feature where we can trigger the bot to start a specific dialog if its mentioned as a parameter in the URL. So something like
https://foo.com/?param=bar
would trigger the bar dialog
This is the code that handles it
function(userId, conversationId, params, token){
return new Promise((resolve, reject)=>{
var _directlineAddress = {
bot: {"id":config.BOT.ID, "name": config.BOT.HANDLE},
channelId: "directline",
serviceUrl: config.BOT.DIRECTLINE_URL,
useAuth: true,
user:{"id": userId},
"conversation": {"id": conversationId}
}
if(params.options){
var _re = /^\?(\w+)*=(\w+)*/
var _programType = _re.exec(params.options);
if (_programType[1] === "foo") {
var _dialogId = "*:/foo";
}
else {
var _dialogId = "*:/" + _programType[1];
}
} else {
var _dialogId = "*:/";
var _specialParams = {"sessionId":token};
}
bot.beginDialog(_directlineAddress, _dialogId, _specialParams, function(err){
else{
resolve();
}
});
})
};
Since i'm splitting the directline from the bot logic , i will no longer be having access to the bot object. therefore bot.beginDialog would not work here
Is there a way i can trigger the dialog by posting to the Directline API?
No. With Direct Line you will be able to send messages to the bot. I guess that a way to go here will be to define a convention message that you will send via Direct Line and that the bot logic will know that it will have to start a dialog based on it.

newbie node.js bot integration with database lookup (looking for best practice)

OK, new to Node.js and botframework. I built first bot using Azure site and downloaded the code. Chose the Luis integrated bot template.
I understand (finally) the event driven model of node.js and the concept of callbacks.
I have the code snippet below. When Luis finds an intent of "Help" it triggers this function. In turn, I have database calls to lookup the entity. Within the database I have an entity, response (if entity is bottle, answer "Recycle").
I have that code working too.
Below the first block is a function handleHelpRequester which is the callback function, I have that working as well. Where I am a little stuck (best practice) is that in this callback function I want to send something to the session object (session.send toward the bottom of the function).
Since I don't create the session object (directly) I'm not sure on the options.
Should I pass the session object to the database function, then pass it back?
Should I create a global variable and set it to the session object (I'm concerned that if multiple people are using this bot then this approach won't work).
I'm open for suggestions.
Thanks
.matches('Help', (session, args) => {
var entities = args.entities;
var itype = builder.EntityRecognizer.findEntity(args.entities, 'ItemTypes');
var respondToUser = '';
var msg = 'Initialized';
// if there is an entity provided, perform a lookup of the entity.
if (itype.entity !== null) {
//Use session.sendTyping so the user thinks something is happening rather than being ignored while we do the lookup in SharePoint.
session.sendTyping();
//perform lookup
respondToUser = sp.lookupEntity(itype.entity, handleHelpRequest);
};
})
function handleHelpRequest(err, respondToUser) {
var msg = 'uninitialized';
if (err = 'success') {
console.log('Respond to user from matches:'.concat(respondToUser));
//return from lookup
if (respondToUser === 'No match found') {
msg = 'I think you are asking for help, but I don\'t understand what you need help with.';
}
else {
msg = 'I can help you with that, \'%s\'.', respondToUser;
}
console.log(msg);
session.send(msg);
//The following two lines are for debugging
session.send('How may I assist you? ' + JSON.stringify(args));
session.send('Value of entity you said: \'%s\'.', itype.entity);
}
else {
console.log('an error occurred');
}
}
If you want to have access to the session object, then pass it as a parameter to your helper function.
Example:
function handleHelpRequest(session, err, respondToUser) {
// do session stuff
}

text integrating api.ai into android

I'm trying to integrate api.ai with android. I have followed the steps required for that. I require integrating text instead of speech. I want to receive the text as input from the user and display it. Can anyone please suggest me the solution for this?
Step 1:Create Configuration for API AI
final AIConfiguration config = new AIConfiguration("<Client access token>",
AIConfiguration.SupportedLanguages.English,
AIConfiguration.RecognitionEngine.System);
aiService = AIService.getService(this, config);
aiService.setListener(this);
aiDataService = new AIDataService(config);
aiRequest = new AIRequest();
Step 2:Set your text here
aiRequest.setQuery(message);
Step 3:Get your response from API AI using AsyncTask
new AsyncTask<AIRequest,Void,AIResponse>(){
#Override
protected AIResponse doInBackground(AIRequest... aiRequests) {
final AIRequest request = aiRequests[0];
try {
final AIResponse response = aiDataService.request(aiRequest);
return response;
} catch (AIServiceException e) {
}
return null;
}
#Override
protected void onPostExecute(AIResponse response) {
if (response != null) {
Result result = response.getResult();
String reply = result.getFulfillment().getSpeech();
sendMessage(message);
mimicOtherMessage(reply);
mListView.setSelection(mAdapter.getCount() - 1);
}
}
}.execute(aiRequest);
If I understand right -- you do not need the Voice input for your Android application and simply want the user to type in the text in their Android application. Post that, you are expecting to pass on that request to API.AI for further processing and a possible response.
If the above is correct, then you do not need any of the Voice capability Activities. Simply take it in the text from the User and pass that on to the API.AI HTTP API.

Resources