I'm handling an intent by playing a MediaObject. I want to create an intent handler that will catch the callback of media play completion, the documentation shows an example on how to write fulfillment code to handle it.
Building your fulfillment
The code snippet below shows how you might write the fulfillment code
for your Action. If you're using Dialogflow, replace
actions.intent.MEDIA_STATUS with the action name specified in the
intent which receives the actions_intent_MEDIA_STATUS event, (for
example, "media.status.update“).
I am confused with the part of the dialogflow instructions. The intent that I handle and return a MediaObject is called smoothie-02 and I have a fallback for it which what gets handled after the media finishes being played, but I want to create another intent and handle it there instead. What I want to do is to create a new intent that would handle it, instead of it going to the fallback intent of smoothie-02 intent.
smoothie-02 handler
app.dialogFlow.intent('smoothie-02', (conv) => {
const welcomeContext = getConvContext(conv, AppContexts.WELCOME);
givenName = welcomeContext.parameters['given-name'];
fruitTypes = welcomeContext.parameters['FruitTypes'];
if (!conv.surface.capabilities.has('actions.capability.MEDIA_RESPONSE_AUDIO')) {
conv.ask('Sorry, this device does not support audio playback.');
return;
}
conv.contexts.set("actions_capability_media_response_audio", 5);
// Zoe says something
let response = `Ooh good choice ${givenName} ! `;
response += fruitTypes.length > 1 ? `${fruitTypes[0]} and ${fruitTypes[1]}` : `${fruitTypes[0]} `;
response += `${drinkType} ` ;
response += 'coming right up. But will you first turn me on?';
console.log(response);
conv.ask(response);
conv.ask(new Suggestions("Don't be shy"));
// Blender plays
conv.ask(new MediaObject({
name: 'Blender Sound',
url: 'https://storage.googleapis.com/zoe-mortimer.appspot.com/blender.wav',
}));
});
What I needed to do is create a new intent and add actions_intent_MEDIA_STATUS in the Events, and that will be the intent that would handle the media playback finished callback. Credits to this article!
Related
I am trying to send a postBack text message to my bot but I don't know the right syntax.
Here is my code:
if (postback.payload == "WHAT_IS_MENTAL_HEALTH") {
await turnContext.sendActivity("TO-DO: Forward on 'What Is Mental Health?' to Bot Handler");
ActionTypes.postBack("What Is Mental Health?");
}
I'm trying to forward on the text "What Is Mental Health?" to my bot so it will pull back the QnA Maker response for that question.
The steps for this are as follows:
User clicks a button on a Facebook Messenger Generic Template Card (e.g. “What Is Mental Health?” Button)
The button sends a postBack payload to the bot (e.g. “WHAT_IS_MENTAL_HEALTH”)
I am detecting the postBack payload from Facebook Messenger (e.g if (postBack.payload == “WHAT_IS_MENTAL_HEALTH”))
Once that particular postBack payload is detected I then want to send an additional postBack to my bot as text (e.g. “What Is Mental Health?”) so it can be interpreted by my QnA and respond with the correct answer which has been programmed into my QnA Knowledge Base.
Facebook Events and the Bot Framework
When Facebook sends an event to your bot, it sends an Activity with an Event ActivityType. For some events, the event data is in the Activity.Value property. For other events, like a PostBack from a Quick Reply, the activity will be a Message and the data will be in Activity.ChannelData. For example, your bot might receive a postBack event as an activity like this:
{
channelId: 'facebook',
[...]
type: 'message',
channelData: {
recipient: {...},
sender: {...},
message: {
[...],
quick_reply: {
[...],
payload: '<your payload>'
}
}
}
}
Handling Facebook Events
This answer is going to pull heavily from the Facebook Events Sample. I highly recommend looking at that for additional help.
Capture Messages and Events
First, you want to capture the facebook messages and events with onMessage() and onEvent():
this.onMessage(async (turnContext) => {
console.log('Processing a Message Activity.');
// Show choices if the Facebook Payload from ChannelData is not handled
if (!await this.processFacebookPayload(turnContext, turnContext.activity.channelData)) {
if (turnContext.activity.channelId !== 'facebook') {
await turnContext.sendActivity('This sample is intended to be used with a Facebook bot.');
}
await this.showChoices(turnContext);
}
});
this.onEvent(async (turnContext) => {
console.log('Processing an Event Activity.');
// Analyze Facebook payload from EventActivity.Value
await this.processFacebookPayload(turnContext, turnContext.activity.value);
});
Process the Messages/Events
Facebook can send many types of events. You may want to use an if or switch statement to handle each type:
async processFacebookPayload(turnContext, data) {
// At this point we know we are on Facebook channel, and can consume the Facebook custom payload present in channelData.
const facebookPayload = data;
if (facebookPayload) {
if (facebookPayload.postback) {
// Postback
await this.onFacebookPostback(turnContext, facebookPayload.postback);
return true;
} else if (facebookPayload.optin) {
// Optin
await this.onFacebookOptin(turnContext, facebookPayload.optin);
return true;
[...]
}
return false;
}
Specifically, Handle a PostBack
The sample just does:
async onFacebookPostback(turnContext, postback) {
console.log('Postback message received.');
// TODO: Your postBack handling logic here...
// Answer the postback and show choices
await turnContext.sendActivity('Are you sure?');
await this.showChoices(turnContext);
}
As you want to route the question to QnA Maker, you might (using the QnA Maker Sample as guidance):
async onFacebookPostback(turnContext, postback) {
// qnaMaker.getAnswers doesn't accept string input, so we need to adjust our turnContext
// to match what it expects, which is a string in Activity.Text
turnContext.activity.text = postback.payload;
const qnaResults = await this.qnaMaker.getAnswers(turnContext);
// If an answer was received from QnA Maker, send the answer back to the user.
if (qnaResults[0]) {
await turnContext.sendActivity(qnaResults[0].answer);
// If no answers were returned from QnA Maker, reply with help.
} else {
await turnContext.sendActivity('No QnA Maker answers were found.');
}
}
'Until loop' analogue needed to continuously read status variable from helper function - and then (when the status variable is 'as we need it') - to resume bot conversation flow.
In my bot (botbuilder v.3.15) I did the following:
During one of my dialogues I needed to open external url in order
to collect some information from the user through that url.
After that I posted collected data (with conversation ID and other info) from that url to my bot app.js
file
After that I needed to resume my bot conversation
For that I created helper file - helper.js in which 'marker' variable is 'undefined' when the data from url is not yet collected, and 'marker' variable is some 'string' when the data is collected and we can continue our bot conversation
helper.js
var marker;
module.exports = {
checkAddressStatus: function() {
return marker;
},
saveAddressStatus: function(options) {
marker = options.conversation.id;
}
}
I can successfully update variable 'marker' with my data, by calling saveAddressStatus function from app.js.
However, when I get back to writing my code which is related to bot conversation flow (To the place in code after which I opened url - in file address.js, and from where I plan to continuously check the 'marker' variable whether it is already updated - in order to fire 'next()' command and continue with session.endDialogWithResult -> and then -> to further bot conversation flows - I cannot find the equivalent of 'until loop' in Node.js to resume my session in bot dialog - by returning 'next()' and continuing with the bot flow.
address.js
...
lib.dialog('/', [
function (session, args, next) {
...
next();
},
function (session, results, next) {
// Herocard with a link to external url
// My stupid infinite loop code, I tried various options, with promises etc., but it's all not working as I expect it
while (typeof helper.checkAddressStatus() == 'undefined') {
console.log('Undefined marker in address.js while loop')
}
var markerAddress = helper.checkAddressStatus();
console.log(markerAddress);
next(); // THE MOST IMPORTANT PART OF THE CODE - IF markerAddress is not 'undefined' - make another step in dialog flow to end dialog with result
function(session, results) {
...session.endDialogWithResult({markerAddress: markerAddress})
}
...
Any ideas how to make a very simple 'until loop' analoque in this context - work?
Having your bot stop and wait for a response is considered bad practice. If all of your bot instances are stuck waiting for the user to fill out the external form, your app won't be able to process incoming requests. I would at least recommend adding a timeout if you decide to pursue that route.
Instead of triggering your helper class in the endpoint you created, you should send a proactive message to the user to continue the conversation. To do this, you will need to get the conversation reference from the session and encode it in the URL that you send to the user. You can get the conversation reference from the session - session.message.address - and at the very least you will need to encode the bot id, conversation id, and the serviceUrl in the URL. Then when you send the data collected from the user back to the bot, include the conversation reference details for the proactive message. Finally, when your bot receives the data, recreate the conversation reference and send the proactive message to the user.
Here is how your conversation reference should be structured:
const conversationReference = {
bot: {id: req.body.botId },
conversation: {id: req.body.conversationId},
serviceUrl: req.body.serviceUrl
};
Here is an example of sending a proactive message:
function sendProactiveMessage(conversationReference ) {
var msg = new builder.Message().address(conversationReference );
msg.text('Hello, this is a notification');
msg.textLocale('en-US');
bot.send(msg);
}
For more information about sending proactive messages, checkout these samples and this documentation on proactive messages.
Hope this helps!
How do I end my conversation from the webhook ?
Marking it within Dialogflow does nothing , basically does not stop it as I am using the webhook for fulfillment .
And if I add it to the code as below then it does not play the media.
// Import the Dialogflow module from the Actions on Google client library.
// https://github.com/actions-on-google/actions-on-google-nodejs
const {dialogflow, Suggestions, MediaObject, Image} = require('actions-on-google');
// Import the firebase-functions package for Cloud Functions for Firebase fulfillment.
const functions = require('firebase-functions');
// Node util module used for creating dynamic strings
const util = require('util');
// Instantiate the Dialogflow client with debug logging enabled.
const app = dialogflow({
debug: true
});
// Do common tasks for each intent invocation
app.middleware((conv, framework) => {
console.log(`Intent=${conv.intent}`);
console.log(`Type=${conv.input.type}`);
//kng
console.log(`Arguments=${conv.arguments}`);
console.log(`Arguments=${typeof(conv.arguments)}`);
// Determine if the user input is by voice
conv.voice = conv.input.type === 'VOICE';
if (!(conv.intent === 'Default Fallback Intent' || conv.intent === 'No-input')) {
// Reset the fallback counter for error handling
conv.data.fallbackCount = 0;
}
});
app.intent('Play Sound', (conv, {SoundType,duration}) => {
const suggestions1 = new Suggestions('do this ', 'do that', 'do nothing');
simple_response = 'this is a response from the webhook'
conv.ask(simple_response)
conv.ask(new MediaObject({
name: SoundType,
url: some_mp3file_url,
icon: new Image({
url: some_image_url,
alt: 'Media icon'
})
}));
conv.ask( suggestions1);
//if I close from the code it doesnot play the sound
conv.close();
//if I comment out the close statement above then it does not close and toggling on the "set this intent as the end of convesation does not seem to help."
}
)
Update - This was intact a bug as pointed out by one of the comments . Reported to google and they fixed the same in April or May
I can duplicate the issue, but it appears to be a bug - playing audio as part of the response and having it close after the audio finishes used to work. It is clearly supposed to be supported - the documentation and the simulator state that Suggestions aren't required if this is a final response.
The workaround is to create an additional Intent that handles the Action actions_intent_MEDIA_STATUS. This Intent would then close the conversation.
After trying and trying countless times, I ask for your help to call a Dialogflow event (GoogleHome) with a specific GoogleHome device.
Through nodeJS I managed to successfully call a Dialogflow event and I get the fullfillment response. All perfect, only I have to let my GoogleHome device speak with fullfillment, I do not need a text-only answer.
My goal is to let my GoogleHome device speak first, without the word "Ok, Google" and wait for a response from the user.
I did not find anything on the web, my attempts stop to invoke the Dialogflow event and have a console response.
This is the code i have tried for fullfillment
test: async function () {
console.log("[funcGHTalk|test] CALLED");
const projectId = "[[projectid]]";
const LANGUAGE_CODE = 'it-IT';
let eventName = "[[eventname]]";
const sessionId = uuid.v4();
const sessionClient = new dialogflow.SessionsClient();
const sessionPath = sessionClient.sessionPath(projectId, sessionId);
// The text query request.
const request = {
session: sessionPath,
queryInput: {
event: {
name: eventName,
languageCode: LANGUAGE_CODE
},
},
};
// Send request and log result
const responses = await sessionClient.detectIntent(request);
console.log('Detected intent');
const result = responses[0].queryResult;
console.log(result);
console.log(` Query: ${result.queryText}`);
console.log(` Response: ${result.fulfillmentText}`);
if (result.intent) {
console.log(` Intent: ${result.intent.displayName}`);
} else {
console.log(` No intent matched.`);
}
}
The code you have written is using the Dialogflow Detect Intent API. This is meant to run on consoles and servers to send a message to Dialogflow, which will parse it, determine which Intent it matches, call fulfillment with that information, and return all the results.
You don't need to run this on a Google Home, since the Google Assistant does all this for you.
What I think you're looking for is to develop fulfillment with Actions on Google and the Dialogflow Fulfillment API. This handles things on the other end - after Dialogflow determines what Intent matches what the user has said, and if that Intent has fulfillment enabled, it will send the information to your webhook which is running on a cloud server somewhere. You would then process it, send a reply (either using the actions-on-google library or the dialogflow-fulfillment library is easiest), and it would send it back to the Assistant.
You indicated that you want the Action to "let my GoogleHome device speak first, without the word "Ok, Google" and wait for a response from the user". This is much more complicated, and not really possible to do with the Google Home device right now. Most Actions have the user initiating the conversation with "Ok Google, talk to my test app" or whatever the name of the Action is.
You don't indicate how you expect to trigger the Home to begin talking, but you may wish to look into notifications to see if those fit your model, however notifications don't work with the Home right now, just the Assistant on mobile devices.
I am trying to build a simple chatbot with DialogFlow.
My aim is to give information from user question, like : where can I slackline above water in croatia ? I have two parameters (croatia, waterline) and a list of slackline places.
So I need a data base to retrieve information from parameters. DialogFlow allows fulfillment with Firebase. I build a database with places (name, country, type of slack) and enable webhook call for my intent.
I use Inline Editor and index.js
const parameters = request.body.queryResult.parameters;
var country = parameters.country.toString();
function show(snap) {
console.log('snap');
agent.add(JSON.stringify(snap.val(),null,2));
}
function slkplc(agent) {
var testRef;
firebase.database().ref('slackplace').once('value',show);
}
// Run the proper function handler based on the matched Dialogflow intent name
let intentMap = new Map();
intentMap.set('slack place', slkplc);
agent.handleRequest(intentMap);
But I do not get the expected result while trying it on DialogFlow or Google Assistant. The function show is asynchronously called but too late and the response is not available for DialogFlow :
I see three way to deal with this problem :
use blocking call to database : another database ?
treat asynchronous message with DialogFlow ???
response to user that an error occured.
The third that I choose, but it is always on error.
After trying several things to wait data from database response, the only thing I managed is to freeze the response, therefore the timeout of DialogFlow - 5s -and Firebase - 60s - were reached.
A workaround
Another way to do it is to separate database acquisition and request/response from DialogFlow. The data of database is collected outside of the dialogflowFirebaseFulfillment
var data;
var inidata = firebase.database().ref().on('value',function(snap) {
console.log('snap');
data = snap.val();
});
exports.dialogflowFirebaseFulfillment = functions.https.onRequest((request, response) => {
const agent = new WebhookClient({ request, response });
...
function slkplc(agent) {
agent.add(JSON.stringify(data,null,2));
}
// Run the proper function handler based on the matched Dialogflow intent name
let intentMap = new Map();
intentMap.set('slack place', slkplc);
agent.handleRequest(intentMap);
}
Now I can do what I want with data, and I am able to find the place where I can practice waterline in croatia. But there is always something weird, the data of the database is duplicated ...
The "right" solution is option 2 that you suggest: since you're doing an asynchronous call, you need to handle this correctly when working with the dialogflow-fulfillment library.
Basically, if your handler makes an asynchronous call, it needs to be asynchronous as well. To indicate to the handleRequest() method that your handler is async, you need to return a Promise object.
Firebase's once() method returns a Promise if you don't pass it a callback function. You can take advantage of this, return that Promise, and also handle what you want it to do as part of a .then() clause. It might look something like this:
function slkplc(agent) {
var testRef;
return firebase.database().ref('slackplace').once('value')
.then( snap => {
var val = snap.val();
return agent.add( JSON.stringify( val, null, 2 ) );
});
}
The important part isn't just that you use a Promise, but also that you return that Promise.