After trying and trying countless times, I ask for your help to call a Dialogflow event (GoogleHome) with a specific GoogleHome device.
Through nodeJS I managed to successfully call a Dialogflow event and I get the fullfillment response. All perfect, only I have to let my GoogleHome device speak with fullfillment, I do not need a text-only answer.
My goal is to let my GoogleHome device speak first, without the word "Ok, Google" and wait for a response from the user.
I did not find anything on the web, my attempts stop to invoke the Dialogflow event and have a console response.
This is the code i have tried for fullfillment
test: async function () {
console.log("[funcGHTalk|test] CALLED");
const projectId = "[[projectid]]";
const LANGUAGE_CODE = 'it-IT';
let eventName = "[[eventname]]";
const sessionId = uuid.v4();
const sessionClient = new dialogflow.SessionsClient();
const sessionPath = sessionClient.sessionPath(projectId, sessionId);
// The text query request.
const request = {
session: sessionPath,
queryInput: {
event: {
name: eventName,
languageCode: LANGUAGE_CODE
},
},
};
// Send request and log result
const responses = await sessionClient.detectIntent(request);
console.log('Detected intent');
const result = responses[0].queryResult;
console.log(result);
console.log(` Query: ${result.queryText}`);
console.log(` Response: ${result.fulfillmentText}`);
if (result.intent) {
console.log(` Intent: ${result.intent.displayName}`);
} else {
console.log(` No intent matched.`);
}
}
The code you have written is using the Dialogflow Detect Intent API. This is meant to run on consoles and servers to send a message to Dialogflow, which will parse it, determine which Intent it matches, call fulfillment with that information, and return all the results.
You don't need to run this on a Google Home, since the Google Assistant does all this for you.
What I think you're looking for is to develop fulfillment with Actions on Google and the Dialogflow Fulfillment API. This handles things on the other end - after Dialogflow determines what Intent matches what the user has said, and if that Intent has fulfillment enabled, it will send the information to your webhook which is running on a cloud server somewhere. You would then process it, send a reply (either using the actions-on-google library or the dialogflow-fulfillment library is easiest), and it would send it back to the Assistant.
You indicated that you want the Action to "let my GoogleHome device speak first, without the word "Ok, Google" and wait for a response from the user". This is much more complicated, and not really possible to do with the Google Home device right now. Most Actions have the user initiating the conversation with "Ok Google, talk to my test app" or whatever the name of the Action is.
You don't indicate how you expect to trigger the Home to begin talking, but you may wish to look into notifications to see if those fit your model, however notifications don't work with the Home right now, just the Assistant on mobile devices.
Related
We are creating a service that will receive WhatsApp messages via the Twilio service. This works, but our issue is that we can't work out how to tell the sender that our server has 'read' the message. The messages always appear as being 'delivered' and never 'read', even after responding to the message. We have looked in the documentation, but can't seem to see how to do this.
Our server is written in NodeJS and is using Express for HTTP side.
Below is an equivalent of the code we are using (not a running example):
import { twiml } from 'twilio';
const { MessagingResponse } = twiml;
async receiveMessage(req: Request, res: Response, next: NextFunction) {
const message = req.body;
// Send back an empty response, we will process asynchronously
const immediateResponse = new MessagingResponse();
res.setHeader('content-type', 'text/xml');
res.send(immediateResponse.toString());
// TODO indicate message as read
// Do what ever logic is needed for given message
const replyMessage = await processMessage(message);
const messageToTwilio = {
body: replyMessage,
from: message.To,
to: message.From
};
const twilioResponse = await this.client.messages.create(messageToTwilio);
// Record value of twilioResponse in DB
}
Can anyone suggest what in the API I should be using for this?
I contacted Twilio on this issue and it turns out this is not currently possible. While they consider this a useful functionality, it is not currently a priority for implementation.
Note, It is possible to get the delivery status of outgoing messages, via the status webhook, but it is not possible to indicate to the remote party that the incoming message was 'read'.
I am building several bots with DialogFlow and Hangouts Chat integration.
How can I retrieve the user email of the user spraking to the bot ?
When I see the request sent from Hangouts chat to Dialogflow I cannot see any information about the user, it's like an anonymous request.
Has anybody find a workaround for that ?
It can be retrieved using events:
For each event like a message, a new user added or removed to the chat you can call event.user and it has the following fields:
event.user.name: The user name
event.user.email: The user email
event.user.displayName: The 'display' user name
event.user.avatarUrl: The user avatar's img url
For example, a working code using onMessage(event) function, for each interaction with the bot, the bot will answer with the user name and the message:
function onMessage(event) {
if (event.type == "MESSAGE" || event.space.type == "DM" ) {
var message = event.user.displayName + " said " + event.message.argumentText;
return { "text": message };
}
}
Response:
Benoit said How to get usernames on Hangouts bots?
A diagram of the JSON event format on Hangouts:
More documentation on hangouts events format and examples of use
Ok. Figured it out...
Indeed, you'll need to handle the intent using a Fullfilment.
In the default fullfilment function, you'll see this bit of code:
exports.dialogflowFirebaseFulfillment = functions.https.onRequest((request, response) => {
const agent = new WebhookClient({ request, response });
console.log('Dialogflow Request headers: ' + JSON.stringify(request.headers));
console.log('Dialogflow Request body: ' + JSON.stringify(request.body));
//... REST OF THE CODE WHERE YOU 'LL HANDLE THE INTENTS
});
The event info that you normally get out of an apps script chat bot is in the
request.body
For example email:
const email = request.body.originalDetectIntentRequest.payload.data.event.user.email;
In that user object, you'll also find:
email
avatarUrl
displayName
Ok, solution here is to enable fulfilment and process this as a cloud function. The input json of the cloud function contains all the event json.
I wanted to reply back "Hello #name" without using cloud function
I created a simple webhook to fulfill a Google Action intent using Actions on Google Client Library. This webhook is hosted on an AWS Lambda function with this code:
'use strict';
// Import the Dialogflow module from the Actions on Google client library.
const {dialogflow} = require('actions-on-google');
// Instantiate the Dialogflow client.
const app = dialogflow({debug: true});
// Handle the Dialogflow intent named 'favorite color'.
// The intent collects a parameter named 'color'.
app.intent('favorite color', (conv, {color}) => {
const luckyNumber = color.length;
// Respond with the user's lucky number and end the conversation.
conv.close('Your lucky number is ' + luckyNumber);
});
// Set the DialogflowApp object to handle the HTTPS POST request.
exports.fulfillment = app;
My issue is the that the response comes back to the assistant in this form:
{
"statusCode": 200,
"body": "{\"payload\":{\"google\":{\"expectUserResponse\":false,\"richResponse\":{\"items\":[{\"simpleResponse\":{\"textToSpeech\":\"Your lucky number is 3\"}}]}}},\"fulfillmentText\":\"Your lucky number is 3\"}",
"headers": {
"content-type": "application/json;charset=utf-8"
}
}
As you can see, the body comes with the escape letter inserted which causes the fulfillment to fail.
I tried the following:
JSON.stringify(conv.close('Your lucky number is ' + luckyNumber));
JSON.parse(conv.close('Your lucky number is ' + luckyNumber));
JSON.parse(conv.close('Your lucky number is ' + luckyNumber).body);
Nothing changed as I think I need to reach the payload part.
Turns out there's an checkbox option in AWS API Gateway called: Use Lambda Proxy Integration.
When selected it returns the JSON as is from my code without extra formatting.
How do I end my conversation from the webhook ?
Marking it within Dialogflow does nothing , basically does not stop it as I am using the webhook for fulfillment .
And if I add it to the code as below then it does not play the media.
// Import the Dialogflow module from the Actions on Google client library.
// https://github.com/actions-on-google/actions-on-google-nodejs
const {dialogflow, Suggestions, MediaObject, Image} = require('actions-on-google');
// Import the firebase-functions package for Cloud Functions for Firebase fulfillment.
const functions = require('firebase-functions');
// Node util module used for creating dynamic strings
const util = require('util');
// Instantiate the Dialogflow client with debug logging enabled.
const app = dialogflow({
debug: true
});
// Do common tasks for each intent invocation
app.middleware((conv, framework) => {
console.log(`Intent=${conv.intent}`);
console.log(`Type=${conv.input.type}`);
//kng
console.log(`Arguments=${conv.arguments}`);
console.log(`Arguments=${typeof(conv.arguments)}`);
// Determine if the user input is by voice
conv.voice = conv.input.type === 'VOICE';
if (!(conv.intent === 'Default Fallback Intent' || conv.intent === 'No-input')) {
// Reset the fallback counter for error handling
conv.data.fallbackCount = 0;
}
});
app.intent('Play Sound', (conv, {SoundType,duration}) => {
const suggestions1 = new Suggestions('do this ', 'do that', 'do nothing');
simple_response = 'this is a response from the webhook'
conv.ask(simple_response)
conv.ask(new MediaObject({
name: SoundType,
url: some_mp3file_url,
icon: new Image({
url: some_image_url,
alt: 'Media icon'
})
}));
conv.ask( suggestions1);
//if I close from the code it doesnot play the sound
conv.close();
//if I comment out the close statement above then it does not close and toggling on the "set this intent as the end of convesation does not seem to help."
}
)
Update - This was intact a bug as pointed out by one of the comments . Reported to google and they fixed the same in April or May
I can duplicate the issue, but it appears to be a bug - playing audio as part of the response and having it close after the audio finishes used to work. It is clearly supposed to be supported - the documentation and the simulator state that Suggestions aren't required if this is a final response.
The workaround is to create an additional Intent that handles the Action actions_intent_MEDIA_STATUS. This Intent would then close the conversation.
I'm handling an intent by playing a MediaObject. I want to create an intent handler that will catch the callback of media play completion, the documentation shows an example on how to write fulfillment code to handle it.
Building your fulfillment
The code snippet below shows how you might write the fulfillment code
for your Action. If you're using Dialogflow, replace
actions.intent.MEDIA_STATUS with the action name specified in the
intent which receives the actions_intent_MEDIA_STATUS event, (for
example, "media.status.update“).
I am confused with the part of the dialogflow instructions. The intent that I handle and return a MediaObject is called smoothie-02 and I have a fallback for it which what gets handled after the media finishes being played, but I want to create another intent and handle it there instead. What I want to do is to create a new intent that would handle it, instead of it going to the fallback intent of smoothie-02 intent.
smoothie-02 handler
app.dialogFlow.intent('smoothie-02', (conv) => {
const welcomeContext = getConvContext(conv, AppContexts.WELCOME);
givenName = welcomeContext.parameters['given-name'];
fruitTypes = welcomeContext.parameters['FruitTypes'];
if (!conv.surface.capabilities.has('actions.capability.MEDIA_RESPONSE_AUDIO')) {
conv.ask('Sorry, this device does not support audio playback.');
return;
}
conv.contexts.set("actions_capability_media_response_audio", 5);
// Zoe says something
let response = `Ooh good choice ${givenName} ! `;
response += fruitTypes.length > 1 ? `${fruitTypes[0]} and ${fruitTypes[1]}` : `${fruitTypes[0]} `;
response += `${drinkType} ` ;
response += 'coming right up. But will you first turn me on?';
console.log(response);
conv.ask(response);
conv.ask(new Suggestions("Don't be shy"));
// Blender plays
conv.ask(new MediaObject({
name: 'Blender Sound',
url: 'https://storage.googleapis.com/zoe-mortimer.appspot.com/blender.wav',
}));
});
What I needed to do is create a new intent and add actions_intent_MEDIA_STATUS in the Events, and that will be the intent that would handle the media playback finished callback. Credits to this article!