so I've been looking into making custom action with API.ai / Dialogflow and I've even tried some things out. One thing I'm a little confused about though is if it's possible to just have an "intent" that just sends whatever you say to a server (one that I would be hosting maybe on a rasp pi) AND can be triggered without having to "run" an app. Most examples have the user say something like "OK google, run mySuperCoolApp" and THEN they can say whatever custom stuff they have set up. I would like to be able to designate some keyword, (maybe "now"?) so that my intent will run, and then send whatever I said to some server I have set up with a webhook.
Example: "OK google, now do task 1"
Then webhook gets triggered (because now is the first word) and sends that string to my server which then does something...
Is this possible?
This is possible, but only locally. ie You could override only on the assistant running locally.
This is done by intercepting the assistant to get the user's query and process it to do local fulfilment.
You could do something like
def process_event(self, event):
if event.type = EventType.ON_RECOGNIZING_SPEECH_FINISHED and event.args:
query = event.args['text']
if query == 'now do task': # check if it matches your command
self.assistant.stop_conversation() # Stop the usual reply from the Assistant
call_a_function() # This will call a function locally to perform a task
Related
I am completely new to the "Actions on Google" world, but following some tutorials (like this) i have already achieved good results.
My test
With Google Assistant and/or Google Home mini send my commands to a personal nodejs online server.
To do this:
i have created a new project on https://console.actions.google.com/
selected conversational option
selected create action / custom intent option
from Dialogflow i have personalized the Default Welcome Intent and created a new Intent with the Fulfillment option set to Enable webhook call for this intent
And obviously, from Dialogflow > Fulfillment, i have enabled the Webhook option (with the url of my nodejs app), and not the Inline editor.
This procedure works, when my app recognizes my custom intent, the answer is sent to my nodejs app online.
My problem
The procedure works, but i always have to do 2 steps before i can perform my action:
1) Hey Google, talk with "nameofmyapp"
2) Say the command
My goal
Execute my command directly without always having to do this 2 steps.
Absolutely! Google calls this "deep linking". With this, you'll be able to do something like
Hey Google, ask nameof myapp to command
See the documentation for details, but in short you'll
Make sure you have an Intent for the command in Dialogflow, with several possible phrases that can be used to trigger it.
These phrases should be what you'd say under "command" in the example above - you'd omit the "to" part.
Go to the Integrations section in Dialogflow, under the Google Assistant integration.
In the Implicit invocation section, select the Intent that you'd like to allow as a deep-linked Intent.
If the command takes action and then should quit, make sure either you have set this in Dialogflow or your fulfillment calls app.close();
So I was using Gmail Google Action on my Assistant device and One thing that I found particularly intriguing was that when a user has given its message and email, after that assistant gives options for change or adding the message. That means on selecting those options, assistant will change the value of message entity or edit it as per user command, without re-enabling the whole intent.
My question is how can I implement this functionality in my Google Action. Is there any particular function made by Google that I can use or do I have to create one from scratch ?
Gmail's integration is a bit different than the way a third-party developer would do it, but you can save user input from the current session and modify it later on in your conversation. In your Action, this wouldn't be just one intent, but a few that would handle the work of modifying some session data and finally using that data to complete the user's original action.
I got started with dialogflow two weeks ago, and have a good basic understanding of how everything works. I'm trying to do the following:
I have one agent, which should take a query from the user (more like a keyword in a sentence, for example "I'd like to buy a phone" or "I'd like to get a loan").
This query should go to a system which has a list of chatbots registered. It has to find the best matched agent for the query.
My question is, how do I redirect from the initial chatbot (which listens to the query) to the 'final' agent?
Is there a blogpost, documentation or something similar for this? I was unfortunately not able to find it.
Thanks in advance!
you can use your base agent's webhook as a dispatcher to relay a message to another agent the same way you would do it with curl.
We took another approach where we're using a middleware in front of DialogFlow that does the dispatching based on a base agent's response.
Once the base agent returns another agent to our middleware, we use the event system to query that specific agent's welcome intent and return it back to the UI.
I have a Actions on Google project that uses api.ai for its actions. This is working well and I can see request/responses appear on the google assistant interface (On mobiles and simulator)
One of my usecases for api.ai needs to broken into 2 parts, in that we have to inform the user that the processing has started and then inform them again once its completed (without them reprompting for the output).
Im trying for a way to inform the user who is using the Google assistant when the processing is completed, but have failed so far. Something like this
User: I would like to see if my loan request is approved
Google Assistant: Hold on, let me check and let u know .
.... (Makes a webservice call to the backend asynchronously)
.... After few seconds ...
.... Postback to google assistant from the webservice
Google Assistant: Thanks for holding, your request is approved.
Im not sure how to do the "postback to google assistant" call. I have tried to get the SessionId from the Api.AI call and then use that to make a event request , but that doesnt seem to send the response to the assistant. Google Assistant seems to be using the formats defined in https://developers.google.com/actions/reference/rest/Shared.Types/AppRequest, but Im unsure how to get the ConversationToken and use that for sending the response back to the user.
Short answer: you can't do that.
Slightly longer answer: At least right now, there is no good way to send a notification. Your Action can only respond to a specific statement from the user. You can say something like "ask again in a minute and I should have a result for you", but that isn't a great experience. At Google I/O 2017, they announced that notifications would be coming to the Google Home at some point... but gave neither a time frame nor any information about an API.
Long, but probably still unsatisfying answer: You can look into Transactions which let them initiate purchase or request of some sort and then "check out". Once they have checked out, you would confirm that a transaction is being processed with an OrderUpdates and then can send updates with the status of the "order". These status updates can turn into notifications or user's can query the state of the order at any time. Transactions don't require payment, so this may work depending on your needs.
However, there are a few things to note. This is still in developer preview, so things may change in the future. It also doesn't work on all surfaces where the Assistant runs, so while it does work on Assistant on phones, it does not work on the Google Home right now.
just getting started with Assistant features in RPi and I am able to successfully implement upto this point and wondering few thing.
Scenario:
user: hey google "please turn on my living room Lights"
List item my code in horword.py : has a function to perform same action based on ON_RECOGNIZING_SPEACH_FINISHED
RPi/google home: I am not sure how respond to that
I was able to capture the request query asked by user using ON_RECOGNIZING_SPEACH_FINISHED = Args.text(str) and use it in my logic to perform the task. However, at the same time, "ok google" is responding with this answer.
to mitigate this problem, I created an google-actions, now it understands my query and respond with intention from api.ai. However, didn't acts on turn lights ON. So, wondering how can I read response from google home/api.ai in text and change code to act on it locally.
appreciate it.
You will not get response as text.
For getting response to client app use webhook in API.AI and send message using fcm to client app.
Read the fcm message in client app and do the corresponding actions.
finally was able to figure out multiple ways. answered this in other stack question. find more details in this post.
Multiple ways to handle this since google doesn't gives voices transcript and we let google say our transcript which is kind off solution for now.