We are planning to support voice action in our application to add user specific action. For example : If user says "Add milk" to google assistant, "milk" is going to added in application db on user account. For that, I have used Account linking(Implicit Flows) in Google Action as per this tutorial (https://developers.google.com/actions/identity/oauth2-implicit-flow).
When i test dialog follow agent, i have enable "Sign-In" required checkbox for all intent including welcome intent. So that, when i invoke my application it will ask the user to sign in to link account to google.
I got the link account message and It has successfully linked. It says "Great! Your account is now linked to Google" message.
Now, I have used webhook url to handle "add " intent. And intent is getting call our api as per the webhook. But when i print header in request, i couldn't see application Access Token, which is saved in Google. But as per the document (https://developers.google.com/actions/identity/oauth2-implicit-flow#handle_data_access_requests) it will be available in header. But i cannot see access token in api request.
Why is the Bearer access token missing in request?
That section gives generic information about the auth token. See the paragraph that says
The specific method of attaching an access token to a request depends on the Google service that's calling your API. See the service's developer documentation for details.
The access token is available in the User object in the accessToken field. You can get the User object using the conv.user object if you're using the actions-on-google library and in the User environment attribute if you're using the multivocal library.
Related
I've signed up for a developer account with Docusign. We have about 15 users that access a Windows Forms page from another program where they fill out the proper information and click a button that calls a NET Core Web API which creates an envelope and sends to the signer. Those 15 users don't have accounts in Docusign and don't need them to. I want to use one admin or service account that will send and receive the emails, but I can't figure out how to do this.
I used my name for the developer account and now all my (test) Docusign emails show they are from me and once the signer signs, my email receives the signed documents. I've tried creating another "Admin" account, but I don't see any way to associate that user as an "API User". When I try using that user's GUID as the ImpersonatedUserID, my RequestJWTUserToken request comes back with "consent_required". But, when I go to the link I just get an error with "The client id provided is not registered with Docusign".
Everything in the Web API is working perfectly except for the emails showing they are from me and then the signed docs coming back to my email. This should be an easy thing to change the user, but apparently it's not, or I just can't figure it out.
You were on your way to fix this, but didn't follow through.
Create new user (you did it)
Find the new userId (you did)
Update JRequestJWTUserToken (you did)
Consent error - expected, the new user did not consent.
Obtain consent. You need to have a URL built correctly like this to do this:
https://account-d.docusign.com/oauth/auth?response_type=code&scope=signature%20impersonation&client_id=7c2b8d7e-xxxx-xxxx-xxxx-cda8a50dd73f&redirect_uri=http://example.com/callback/
Once you do 5, you need to log in with the same new user account you created (you may be automatically logged in with your original developer account, so log out) and provide consent.
Then step 4 will work just fine.
https://developers.docusign.com/platform/auth/consent/obtaining-individual-consent/
My goal is to use the Facebook webhooks to get a notification whenever an IG user mentions my IG account. For that I have followed these steps:
I have created a Facebook page and connected it with an Instagram business account.
I have created a Facebook app (which is now live) and added the product "webhooks".
I have connected my app with my page.
I have created a webhook and subscribed to the mentions event.
I have tested the webhook, using the tools provided by Facebook and it works.
The problem I'm facing is that I can't get my app to work with real data. I have tried to mention my business IG account from my IG personal account, but nothing happens. My callback url is not getting called.
My app doesn't need users to login, so I'm not sure if I need to send my app for review. Do I need a special permission?
I had the same problem. For my case, when I installed the app via graph api explorer, the page token I was using didn't have the instagram_manage_insights permission. So I granted that permission, got a new token, re did the POST request to the {page-id}/subscribed_apps again with the new token and that fixed the problem. Just for the record, I used a system user token from the business manager.
Did you verified your Facebook app via app review? If not, you won't get production data via a webhook callback for the Instagram. I had the same issue and asked the Facebook support and got following response:
However, this is not the same for instagram or pages. This is just how the product teams have decided to implement it. For app review, you can show a mock process of the flow, using either the test webhook or your own process. The app review is less about technical implementation steps, but just a way to make sure that your app is going to use the permission in a way that follows our guidelines, so mocking the procedure should be fine. The reviewers understand that you do not receive webhooks in dev mode and should take this into consideration.
For more information check out following links:
Why is the Instagram Graph API webhook not working
https://developers.facebook.com/support/bugs/495933900986533/
I just figure it out of this problem.
There is a mistake in the official Instagram webhook guide:
With Graph API version 3.2, the /{page-id}/subscribed_apps edge now requires the subscribed_fields parameter, which currently does not support Instagram webhooks fields. To get around this, use your app's dashboard to subscribe.
https://developers.facebook.com/docs/instagram-api/guides/webhooks/#install-app
However, the app's dashboard subscribe is another function. You need your page to subscribe to some field to receive webhooks
Just POST /{your-instagram-connected-page-id}/subscribed_apps?subscribed_fields=feed&access_token={your-instagram-connected-page-token}
Then you will receive webhooks in live mode.
Update: Having my own OAuth server:
Thanks for sharing the step-by-step instructions link. Also, I don't have firebase hosting.
Following is my understanding, please correct me wherever I am wrong:
Approach 1
I will need to activate firebase hosting and build 2 endpoints. One for authorization exchange and another for token exchange.
I will need to use an OAuth server. Let say: ory-hydra and configure it with the endpoints I created in firebase. Or AWS Lambda.
Host the OAuth (ory-hydra) server somewhere on the internet.
Use these endpoints on actions on google and make a simple webpage where users will be redirected to authenticate.
Approach 2
I will need to activate firebase hosting and build 2 endpoints. One for authorization exchange and another for token exchange.
Use the firebase functions to implement OAuth and token endpoints. (I am not sure how to do this and if its possible)
Use these endpoints on actions on google and make a simple webpage where users will be redirected to authenticate.
Please correct me if I am wrong.
Update: After making the following changes:
Changed the authorization URL to:
https://accounts.google.com/o/oauth2/v2/auth
Changed the authorization type to Implicit.
Now, I am getting the authorization URL in the debug section, and I am able to authorize by pasting that URL in another tab.
However, I am still facing issues in getting user information. I have following code in the input.welcome intent:
'input.welcome': () => {
// Use the Actions on Google lib to respond to Google requests; for other requests use JSON
if (requestSource === googleAssistantRequest) {
sendGoogleResponse('Hello, Welcome to my First Fulfillment agent!'); // Send simple response to user
app.askForSignIn();
let displayName = app.getUserName().displayName;
console.log(displayName)
}
All I am getting is Null in the debug logs.
Previous Question I am trying to get user's information in the google action intent. Following is what I did:
1.) Created an OAUTH key for my project from google developer console. https://console.developers.google.com/apis/dashboard?project=.
2.) Logged into console.actions.google.com and clicked on Account Linking.
3.) Entered the information. Please refer the screenshot to see the information I entered.
4.) In my 'input.welcome' intent added following code:
app.askForSignIn();
Now in the simulator, I am getting: "It looks like your my test app account is not linked yet." I have also checked sign-in required for my intent from the dialog flow UI.
Previous question
I am trying to send an email from my google assistant conversational bot. I am able to send emails to using nodemailer. However, I am not able to get the user email address.
Following is my code to send email:
var transporter = nodemailer.createTransport({
service: 'gmail',
auth: {
user: 'My-Email',
pass: 'My-Pass',
}
});
var mailOptions = {
from: 'Sender-Email',
to: 'Receiver-Email',
subject: 'Requested Information',
text: 'Your information is here'
};
function sendEmail(){
transporter.sendMail(mailOptions, function(error, info){
if (error) {
console.log(error);
} else {
console.log('Email sent: ' + info.response);
}
});
}
And finally, I am calling it in the action intent:
'input.sendmail': () => {
sendEmail();
},
Till this point I am able to send emails when someone says, send email to my google assistant action.
After this, I tried to get the user's email address using the following methods:
const app = new DialogflowApp({request: request, response: response});
console.log(app.getUserName())
console.log(app.getUser().userName)
console.log(app.getUser().userId)
But none of them gave me user's information. Instead, I am getting following information in the dialogflow console log:
{
userStorage: '{"data":{}}',
lastSeen: '2018-03-05T10:18:17Z',
locale: 'en-US',
userId: 'ABadfdfrffsdffNa0H4hlCy_eyZmVNa8LweMJMCyirUg-
qAx8FHwvSI49QurUhxhgLsT6IUU4nGfF1',
user_id: 'ABerysteui4hlCy_eyZmVNa8LweMJMCyirUg-
qAx8FHwvSI49QurUhxhgLsT6IUU4nGfF1',
access_token: undefined,
userName: null
}
I tried to google this issue and it seems I will need to follow [1], not sure though.
I will appreciate if someone can tell me if [1] is the correct guide to follow, or I will need to something else?
Thanks!
[1] https://developers.google.com/actions/identity/account-linking
That is correct. For security and privacy purposes, there is no way to get the email address associated with the account used to setup the Assistant account.
The correct way to go about this would be to implement account linking. With this, you would create an account on your system and, as part of that account, get the user's email address (typically via an app or webapp). The account linking would then connect your account to the Assistant's account and, when they use that account to access your Action, you'd be sent an access token which you can use to identify which of your accounts this is. You can then get the user's information from your account info.
When the user is prompted to link their account for the first time, Google Home users will get an activity card on the Google Home app on their phone which will direct them to your auth page. From a mobile device, it should open in the Assistant directly.
If you do not already have accounts or a login page, you should be able to build this with Firebase Authentication and the Google auth provider.
Update for clarity: To be clear - just linking your account to their Assistant account won't automatically give you the information about their Assistant account. You can get their email address (your original request) when they setup the account with you by requesting the profile scope as part of the OAuth. You can then use the information you've collected about them when you know they have connected to you via the Assistant.
In your updated question, you're trying to get their name after they have logged in through the Assistant. If all you wanted was their name, you could have asked for permission to get this without requiring Account Linking or login. (Or, as noted above, you could have asked for this when they created the account with you.)
Update (Based on your question about OAuth)
Two things to your update.
Being prompted "It looks like your account isn't linked yet" is normal. You'll need to use the URL provided in the response tab to continue the account linking.
More significantly, however, it doesn't sound like you've setup an OAuth server - just that you're trying to configure things. The screen shot make it look like you're just using the URL that is supposed to be to respond with auth tokens - not where the system will go to request them. Make sure you have read the documentation at https://developers.google.com/actions/identity/account-linking to see what values should be in the configuration and what other tasks you need to do.
Update based on your comment that you need an OAuth server.
Yes, you need an OAuth server. You cannot just use Google's, even if you just want your users to log into their Google account.
There are quite a few OAuth servers available, however a google search for "open source openid connect server" or "open source oauth server" shows some promising results.
Additionally, Google gives you step-by-step instructions on what it is expecting for an OAuth server it connects to. So you would need to
Implement a way for a user to create and log into an account on your service and
Implement the OAuth exchange protocols as Google has described (I suggest the Auth Code Flow method).
You do not need your own domain - you can implement both of these through Firebase Functions and Firebase Hosting which includes a SSL certificate for a hostname for your project.
Update addressing your possible approaches to implementing an OAuth2 server.
First of all - you don't need to use Firebase Hosting and/or Firebase Functions for anything. They're just an option that provide you a valid HTTPS endpoint.
Approach 1 - use an external package such as ory-hydra
I'm not sure what the point of the Firebase Hosting would be in your example. The auth exchange endpoint and token exchange endpoint are exactly what the OAuth2 server is there to do.
I don't know much about ory-hydra, but it certainly seems a reasonable solution. You would need to host it somewhere (AWS, Google Compute Engine, or other hosting provider that would work with it), but it should provide the endpoints you need. From a quick reading of ora-hydra, you will need to provide an account backend of some sort and a way for your users to login to that account.
Approach 2 - implement using Firebase
You have this completely correct. It is fairly straightforward (not necessarily easy - but straightforward) to do a simple OAuth2 implementation with Firebase Cloud Functions combined with a login page hosted on Firebase Hosting that uses Firebase Authentication for the login.
Which approach you take is up to you. Using an existing solution is certainly easier, hopefully more reliable and secure, and will let you focus on the Action more itself, but may still require a lot of integration work. Implementing an OAuth2 server will give you a better understanding of OAuth2, but runs a higher risk of problems.
I am able to make it work after a long time. We have to enable the webhook first and we can see how to enable the webhook in the dialog flow fulfillment docs If we are going to use Google Assistant, then we have to enable the Google Assistant Integration in the integrations first. Then follow the steps mentioned below for the Account Linking in actions on google:-
Go to google cloud console -> APIsand Services -> Credentials -> OAuth 2.0 client IDs -> Web client -> Note the client ID, client secret from there -> Download JSON - from json note down the project id, auth_uri, token_uri -> Authorised Redirect URIs -> White list our app's URL -> in this URL fixed part is https://oauth-redirect.googleusercontent.com/r/ and append the project id in the URL -> Save the changes
Actions on Google -> Account linking setup 1. Grant type = Authorisation code 2. Client info 1. Fill up client id,client secrtet, auth_uri, token_uri 2. Enter the auth uri as https://www.googleapis.com/auth and token_uri as https://www.googleapis.com/token 3. Save and run 4. It will show an error while running on the google assistant, but dont worry 5. Come back to the account linking section in the assistant settings and enter auth_uri as https://accounts.google.com/o/oauth2/auth and token_uri as https://accounts.google.com/o/oauth2/token 6. Put the scopes as https://www.googleapis.com/auth/userinfo.profile and https://www.googleapis.com/auth/userinfo.email and weare good to go. 7. Save the changes.
In the hosting server(heroku)logs, we can see the access token value and through access token, we can get the details regarding the email address.
Append the access token to this link "https://www.googleapis.com/oauth2/v1/userinfo?access_token=" and we can get the required details in the resulting json page.
`accessToken = req.get("originalRequest").get("data").get("user").get("accessToken")
r = requests.get(link)
print("Email Id= " + r.json()["email"])
print("Name= " + r.json()["name"])`
I have AWS Cognito account linking setup with Actions on Google. I have requested the scope of email. I am using API.AI, how can I get the user's email? From what I've read, you get a userId that is basically an anonymous id. How can I get user details?
For getting user details like mail id etc, you would have to enable Account linking on your google home dialog flow console.
You would also have to provide an Oauth2 server to get the authorization done.
Refer to this : https://developers.google.com/actions/identity/account-linking
Once this is done, you will receive an auth_token every time your api.ai app is invoked. You will then have to use this token and retrieve the user details like mail id from Google API services.
Let me know if I need to add more details, in case its not clear.
I want to recieve the user email using Google Actions as documented here but docs talks about EMAIL permission, but when I read the permission docs here I can't find any EMAIL permission. Any help? How can I receive the user email?
This is WORKING ,you can do this with account linking.
We have to enable the webhook first and we can see how to enable the webhook in the dialog flow fulfillment docs
If we are going to use Google Assistant, then we have to enable the Google Assistant Integration in the integrations first.
Then follow the steps mentioned below for the Account Linking in actions on google:-
Go to google cloud console
goto API's and Services -> Credentials -> OAuth 2.0 client IDs -> Web client
Note the client ID, client secret from there
Download JSON - from json note down the project id, auth_uri, token_uri
goto Authorised Redirect URIs
White list our app's URL, in this URL fixed part is https://oauth-redirect.googleusercontent.com/r/[project-Id] (replace [project-Id] with your project id)
Save the changes
Go to Actions on Google(https://console.actions.google.com) -> Account linking setup
select Grant type = Authorisation code
Client info
Fill up client id,client secrtet, auth_uri, token_uri
Enter any random url as Authorization URL and token_uri such as https://example.com/auth and https://example.com/token
Save
It will show an error while running on the google assistant, but dont worry
Come back to the account linking section in the assistant settings and this time enter correct auth_uri as https://accounts.google.com/o/oauth2/auth
and token_uri as https://accounts.google.com/o/oauth2/token
note that it is some sort of problem in from their side that not allows you to use this url in first hit and will keep saying "Generic URLs are not allowed. You must provide a valid token url specific to your Assistant app." so just give any random url in first hit and save, then comeback again it will allow you these urls :-)
Put the scopes as https://www.googleapis.com/auth/userinfo.profile and https://www.googleapis.com/auth/userinfo.email
and weare good to go.
Save the changes.
In the hosting server logs, we can see the access token value and through access token, we can get the details regarding the email address.
Append the access token to this link "https://www.googleapis.com/oauth2/v1/userinfo?access_token=" and we can get the required details in the resulting json page.
write this code
accessToken = req.get("originalRequest")
.get("data")
.get("user")
.get("accessToken")
r = requests.get(link) // make get request to link
print("Email Id: " + r.json()["email"])
print("Name: " + r.json()["name"])
Ya, unfortunately the Assistant's SDK doesn't give you the email address. But if you implement account linking (like Ahmed mentioned) and use the Streamlined Flows, then you'll be getting the email provided to you; you just need to use the jsonwebtoken library and you can decode the assertion JWT and grab the email address.
That being said, this happens during "sign in" and token exchange... not during the actual action fulfillment. You'll need to issue a refresh token / access token :S
One approach is to go with account linking. I'm wondering what use cases you might have that won't necessarily work without email or account linking?