Access Facebook Messenger User Profile API in DialogFlow - dialogflow-es

I'm building a cross-platform chatbot in Google's DialogFlow. I'd like to access the Facebook User Profile API to learn the user's first name.
I'm struggling to find advice on how (or if) I can make this happen.
https://developers.facebook.com/docs/messenger-platform/identity/user-profile/
Has anybody here achieved this?

I did that for one of my bots yesterday, you need 2 things, first the Page Token and second is the psid which is Page scope user ID.
On dialogflow, you will receive the request block with psid as sender id. You can find it at:
agent.originalRequest.payload.data.sender.id
This psid needs to be passed to api get request at
/$psid?fields=first_name with your page Token as accessToken to get the first name in response.

You need to make a call to Facebook Graph API in order to get user's profile.
Facebook offers some SDKs for this, but their official JavaScript SDK is more intended to be on a web client, not on a server. They mention some 3rd party Node.js libraries on that link. I'm particularly using fbgraph (at the time of writing, it's the only one that seems to be "kind of" maintained).
So, you need a Page Token to make the calls. While developing, you can get one from here:
https://developers.facebook.com/apps/<your app id>/messenger/settings/
Here's some example code:
const { promisify } = require('util');
let graph = require('fbgraph'); // facebook graph library
const fbGraph = {
get: promisify(graph.get)
}
graph.setAccessToken(FACEBOOK_PAGE_TOKEN); // <--- your facebook page token
graph.setVersion("3.2");
// gets profile from facebook
// user must have initiated contact for sender id to be available
// returns: facebook profile object, if any
async function getFacebookProfile(agent) {
let ctx = agent.context.get('generic');
let fbSenderID = ctx ? ctx.parameters.facebook_sender_id : undefined;
let payload;
console.log('FACEBOOK SENDER ID: ' + fbSenderID);
if ( fbSenderID ) {
try { payload = await fbGraph.get(fbSenderID) }
catch (err) { console.warn( err ) }
}
return payload;
}
Notice you don't always have access to the sender id, and in case you do, you don't always have access to the profile. For some fields like email, you need to request special permissions. Regular fields like name and profile picture are usually available if the user is the one who initiates the conversation. More info here.
Hope it helps.
Edit
Promise instead of async:
function getFacebookProfile(agent) {
return new Promise( (resolve, reject) => {
let ctx = agent.context.get('generic');
let fbSenderID = ctx ? ctx.parameters.facebook_sender_id : undefined;
console.log('FACEBOOK SENDER ID: ' + fbSenderID);
fbGraph.get( fbSenderID )
.then( payload => {
console.log('all fine: ' + payload);
resolve( payload );
})
.catch( err => {
console.warn( err );
reject( err );
});
});
}

Related

Xero API 403 - AuthenticationUnsuccessful error

I am using the Xero Api with Nodejs and the xero-node library.
I have completed the oAuth flow and saved the token to the database. The issue i am now having is continually getting a 403 forbidden error when attempting to get anything from Xero be that Contacts, Accounts or Users. Sample code is below
I can get tenants ok without an issue however anything else doesn't work. I have checked the scopes to make sure when I am setting up the client they are correct which they are.
var getStuff = async(tokenSet) => {
await xero.setTokenSet(tokenSet);
const tenants = await xero.updateTenants();
const xeroTenantId = tenants[0].id // {String} Xero identifier for Tenant
const ifModifiedSince = new Date("2020-02-06T12:17:43.202-08:00");
const where = 'IsSubscriber==true'; // {String} Filter by an any element
const order = 'LastName ASC'; // {String} Order by an any element
console.log(tenants);
try {
const response = await xero.accountingApi.getUsers(xeroTenantId, ifModifiedSince, where, order);
console.log(response.body || response.response.statusCode)
}
catch (err) {
/// console.log(err);
console.log(`There was an ERROR! \n Status Code: ${err.response.statusCode}.`);
console.log(`ERROR: \n ${JSON.stringify(err.response.body, null, 2)}`);
}
}
Which scopes have been added to the access token you are passing through? You can decode your token here https://jwt.io/
Also - you need to pass the ‘tenant.tenantId’ to the function. I believe the tenant.id actually relates to the connection id which is coming back from the /connections endpoint.
My hunch is that is the issue. Also curious how that’s working, as updateTenants() should have the empty function call on the end. Is that working for you?

How can I create an entity specific to a user?

I'm creating an action for Google Assistant with Dialogflow and actions-on-google-nodejs that accesses the GitKraken Glo API to add cards to people's boards. I'm authenticating my users with Account Linking. I want my users to be able to say things like Add a card to [board name] or Add a card. If a board name isn't given I want the action to prompt the user for it. How can I create a session entity that get's all the board names for the logged in user?
Sorry if this doesn't make much sense, I'm pretty new to Actions on
Google and Dialogflow. Feel free to ask questions for clarity.
There are a few things you'll need to do first to use a Session Entity:
The Entity Type needs to already exist. Session Entities update existing ones. The easiest way to do this is to create the Entity you want in the Dialogflow UI. It doesn't need to have any Entities in it, but having one as a default can be useful.
You need a Service Account for your project in Google Cloud that will do the update, and a secret key for this account.
Your life will be a lot easier if you use a library, such as the dialogflow-nodejs library.
In general, your code needs to do the following, typically when the user first starts the session (ie - in your Welcome Intent Handler):
Get the list of boards
Update the Session Entity Type, creating an Entity for each board. Doing this update involves:
Issuing a patch against the projects.agent.sessions.entityTypes method with a SessionEntityType for the Entity Type you're overriding.
The SessionEntityType will contain an array of Entities with the canonical name (likely the board name, or some unique identifier) and any aliases for it (at least the board name, possibly anything else, possibly including aliases like "the first one" or "the most recent one").
The README for the library includes links to sample code about how to do this using the nodejs library. Code that I have that does this work has a function like this:
function setSessionEntity( env, entityType ){
const config = envToConfig( env );
const client = new dialogflow.SessionEntityTypesClient( config );
let parent = env.dialogflow.parent;
if( entityType.displayName && !entityType.name ){
entityType.name = `${parent}/entityTypes/${entityType.displayName}`;
}
if( !entityType.entityOverrideMode ){
entityType.entityOverrideMode = 'ENTITY_OVERRIDE_MODE_OVERRIDE';
}
const request = {
parent: parent,
sessionEntityType: entityType
};
return client.createSessionEntityType( request );
}
conv.user.email
You can use conv.user object :
const Users = {};
app.intent('Get Signin', (conv, params, signin) => {
if (signin.status === 'OK') {
const email = conv.user.email
Users[email] = { };
conv.ask(`I got your email as ${email}. What do you want to do next?`)
} else {
conv.ask(`I won't be able to save your data, but what do you want to next?`)
}
})
app.intent('actions.intent.TEXT', (conv, input) => {
if (signin.status === 'OK') {
Users[conv.user.email] = {
lastinput: input
};
}
});
conv.id
Also with conv id is unique id for the current conversation.
// Create an app instance
const app = dialogflow()
// Register handlers for Dialogflow intents
const Users = {};
app.intent('Default Welcome Intent', conv => {
Users[conv.id] = {
conversationId: conv.id,
name: '1234'
};
})
app.intent('actions.intent.TEXT', (conv, input) => {
Users[conv.id] = {
lastinput: input
};
});
app.intent('Goodbye', conv => {
delete Users[conv.id];
})

Dialogflow Webhook V2 Account linking cannot get user profile info

I have managed to make Account Linking on Android using actions on google sdk and Dialogflow but can't make it work using the new Dialogflow V2 webhook.
It does ask permission to link account and the account linking does happen but I cannot access the user data like given_name, email, etc.
I have done everything required including creating the intent with actions_intent_SIGN_IN which gets called as expected except that I don't get the information that I should.
This is the code that is called to create sign in
function googleSignIn() {
console.log('In googleSignIn')
const conv = agent.conv()
// conv.ask(new SignIn('To get your account details'))
// conv.ask('Hello')
var payload = {
"expectUserResponse": true,
"systemIntent": {
"intent": "actions.intent.SIGN_IN",
"data": {}
}
}
// agent.add(payload)
console.log('add googleSignIn payload')
agent.add('Placeholder_google_payload')
agent.add(new Payload(agent.ACTIONS_ON_GOOGLE, payload))
}
This gets called when the intent is fired by Actions on Google
function getSignIn(agent) {
console.log('In function getSignin: ')
const conv = agent.conv()
console.log('conv parts: ', JSON.stringify(conv.user))
agent.add('User profile:', conv.user);
// console.log('conv access token: ', JSON.stringify(conv.user.access.token))
}
With help from Prisoner I managed to solve the problem. You basically just get the token which has to be decoded. This is the relevant code.
var jwt = require('jsonwebtoken'); // use npm i jsonwebtoken to get
const profile = jwt.decode(conv.user.profile.token);
console.log('Profile: ', profile);
// you can then get the elements from the profile (instead of payload)
// e.g.
console.log('Name: ', profile.name);
Once the user has signed in once using Google Sign In for Assistant, all further requests will contain an id token.
If you are using the actions-on-google library, the token will be in the conv object as conv.user.profile.id and the profile information itself (the "payload" of the id token) will be in conv.user.profile.payload.
If you're using the dialogflow-fulfillment library, you can get the conv object from agent.getConv().
If you're using the multivocal library, the user profile is stored in the environment under User/Profile with the id token available under User/IdentityToken and User/IsAuthenticated will be set to true.
This will be sent to you for this session, and any future session as long as the user doesn't disconnect your Action from their account.

Spotify node web api - trouble with multiple users

I am working on an app that uses Spotify Node web API and having trouble when multiple users login into my application. I am successfully able to go through authentication flow and get the tokens and user ID after a user logs in. I am using the Authorization Code to authorize user (since I would like to get refresh tokens after expiration). However, the current problem is that getUserPlaylists function described here (FYI, if the first argument is undefined, it will return the playlists of the authenticated user) returns playlists of the most recently authenticated user instead of the user currently using the app.
Example 1: if user A logins in to the application, it will get its playlists fine. If user B logins in to the application, it also sees its own playlists. BUT, if user A refreshes the page, user A sees the playlists of the user B (instead of its own, user A playlists).
Example 2: user A logs in, user B can see user A's playlists just by going to the app/myplaylists route.
My guess is, the problem is with this section of the code
spotifyApi.setAccessToken(access_token);
spotifyApi.setRefreshToken(refresh_token);
The latest user tokens override whatever user was before it and hence the previous user is losing grants to do actions such as viewing its own playlists.
Expected behavior: user A sees own playlists after user B logs in event after refreshing the page.
Actual behavior: user A sees user B's playlists after user B logged in and user A refreshes the page.
I am aware that I could use the tokens without using the Spotify Node API
and just use the tokens to make requests and it should probably be fine, however, it would be great to still be able to use the Node API and to handle multiple users.
Here is the portion of code that most likely has problems:
export const createAuthorizeURL = (
scopes = SCOPE_LIST,
state = 'spotify-auth'
) => {
const authUrl = spotifyApi.createAuthorizeURL(scopes, state);
return {
authUrl,
...arguments
};
};
export async function authorizationCodeGrant(code) {
let params = {
clientAppURL: `${APP_CLIENT_URL || DEV_HOST}/app`
};
try {
const payload = await spotifyApi.authorizationCodeGrant(code);
const { body: { expires_in, access_token, refresh_token } } = payload;
spotifyApi.setAccessToken(access_token);
spotifyApi.setRefreshToken(refresh_token);
params['accessToken'] = access_token;
params['refreshToken'] = refresh_token;
return params;
} catch (error) {
return error;
}
return params;
}
export async function getMyPlaylists(options = {}) {
try {
// if undefined, should return currently authenticated user
return await spotifyApi.getUserPlaylists(undefined, options);
} catch (error) {
return error;
}
}
Would appreciate any help on this. I am really excited about what I am making so it would mean a LOT if someone could help me find the issue...
You're on the right track. When you set your access token and refresh token, though, you're setting it for your entire application, and all users who call your server will use it. Not ideal.
Here's a working example of the Authorization Code Flow in Node: https://glitch.com/edit/#!/spotify-authorization-code
As you can see, it uses a general instance of SpotifyWebApi to handle authentication, but it instantiates a new loggedInSpotifyApi for every request to user data, so you get the data for the user who's asking for it.
If you want to use the above example, you can just start editing to "remix" and create your own copy of the project.
Happy hacking!

Authorising a Spotify session on a headless system

Clearly by the negative score, I haven't provided enough information - sorry about that. However, perhaps add comments to explain why rather than just marking it down?
2nd attempt at a description:
I would like to be able to connect to Spotify's web API interface (https://developer.spotify.com/web-api/) on a headless embedded platform (Arm based simple MCU with WiFi). The username and password would be hardcoded into the system, probably added at setup time with the help of a mobile device (providing a temporary user interface).
I want to be able to add tracks to a playlist, which requires an authentication token. Spotify's usual flow requires the embedded platform to host their webpage login, as described here (https://developer.spotify.com/web-api/authorization-guide/).
Is this possible to authenticate without the webpage?
I have seen here (https://developer.spotify.com/technologies/spotify-ios-sdk/token-swap-refresh/) that Spotify recommend mobile apps use a remote server to handle refreshing of tokens - perhaps that's a route?
Any pointers would be appreciated.
I don't think it is bad question. I am also working on a headless player that runs on a local network which makes the authorization flow a bit awkward. So this is not much of an answer, but let me explain how it can be done.
Your headless system needs to have a web interface that can redirect to the spotify authorization url and handle the callback. The problem is that you have to register the callback url on your spotify app. Say you register http://server1/spotify/auth/callback. Now the server1 needs to be accessible from the device doing the authorization, f.ex by adding it to /etc/hosts.
The good news is that refresh can be done without user intervention, so if you store the access token the user will only need to do this one time after installing.
I know that this is really late, but for anyone having the same issue...
I am working on something similar was mentioned above so I'll share what I know. I am creating a music player that could act as another device on my Spotify (using: https://developer.spotify.com/documentation/web-playback-sdk/) account as well be controlled by my custom webpage.
I have 3 parts to this: backend server, the SDK player webpage (for me: http://localhost:8080/#/pup/player), the frontend UI webpage
(all the code snippets are a part of a class)
The only way I was able to get it running was like so:
Start the backend server and initialize puppeteer
async initPup(){
this.browser = await puppeteer.launch({
headless: false, // This is important, because spotify SDK doesn't create the device when using headless
devtools: true,
executablePath: "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe", //I also have to use Chrome and not Chromium, because Chromium is missing support for EME keySystems (yes, I've tried bruteforcing chromium versions or getting Firefox to work using createBrowserFetcher())
ignoreDefaultArgs: ['--mute-audio'],
args: ['--autoplay-policy=no-user-gesture-required']
});
this.page = (await this.browser.pages())[0]; // create one page
if(this.page == undefined){
this.page = await this.browser.newPage();
}
this.pup_ready = true;
console.log(await this.page.browser().version())
}
Open your SDK player page with puppeteer and pass the ClientID and ClientSecret of your Spotify project (https://developer.spotify.com/dashboard/):
async openPlayer(){
// const player_page = "http://localhost:8080/#/pup/player"
if(this.pup_ready){
await this.page.goto(player_page + "/?&cid=" + this.client_id + "&csec=" + this.client_secret);
}
}
On the SDK player webpage save the cid and csec URL params to LocalStorage. This should be done when no ULR parameter named "code" has been given, because that's the authorizations code which will be generated in the next step.
Something like:
var auth_code = url_params_array.find(x=>x.param.includes("code")); // try to get the auth code
var c_id = url_params_array.find(x=>x.param.includes("cid")); //get cid
var c_sec = url_params_array.find(x=>x.param.includes("csec")); //get csec
var token = undefined;
if(auth_code == undefined){ // the auth code is not defined yet and it has to be created
//SAVING CLIENT ID and CLIENT SECRET
c_id = c_id.value;
c_sec = c_sec.value;
window.localStorage.setItem("__cid", c_id)
window.localStorage.setItem("__csec", c_sec)
//GETTING THE AUTH CODE
var scope = "streaming \
user-read-email \
user-read-private"
var state = "";
var auth_query_parameters = new URLSearchParams({
response_type: "code",
client_id: c_id,
scope: scope,
redirect_uri: "http://localhost:8080/#/pup/player/",
state: state
})
window.open('https://accounts.spotify.com/authorize/?' + auth_query_parameters.toString()); // tak the puppeteer to the spotify login page
}
Login on the spotify page using your credential to create the auth token. I had to use https://www.npmjs.com/package/puppeteer-extra-plugin-stealth to bypass CAPTCHAS
async spotifyLogin(mail="<YOUR_SPOTIFY_MAIL>", pass = "<YOUR_SPOTIFY_PASSWORD") {
var p = this.page = (await this.browser.pages())[1] // get the newly opened page with the spotify
//await p.waitForNavigation({waitUntil: 'networkidle2'})
await p.focus("#login-username"); // put in the credentials
await p.keyboard.type(mail);
await p.focus("#login-password");
await p.keyboard.type(pass);
await p.$eval("#login-button", el => el.click());
(await this.browser.pages())[0].close(); // close the old SDK page
await sleep(1000) // wait to be redirected back to your SDK page
//
this.page = (await this.browser.pages())[0];
this.auth_code = await this.page.evaluate( (varName) => window.localStorage.getItem(varName), ["__auth"] ) // here is ave the auth token as a property of the class instance as well
}
Once you're redirected to SDK page again you already have cid and csec and now also the auth token.
if(auth_code == undefined)
//... (this is already in step 3)
}else{
// GETTING CID and C SECRET AGAIN
c_id = window.localStorage.getItem("__cid")
c_sec = window.localStorage.getItem("__csec")
// SAVING THE AUTH CODE
auth_code = auth_code.value;
window.localStorage.setItem("__auth", auth_code)
}
Generate a token on the backend.
async genToken():Promise<void>{
//Pretty much coppied from: https://developer.spotify.com/documentation/web-playback-sdk/guide/
var authOptions = {
url: 'https://accounts.spotify.com/api/token',
headers: {
'Authorization': 'Basic ' + (Buffer.from(this.client_id + ':' + this.client_secret).toString("base64"))
},
form: {
code: this.auth_code,
redirect_uri: "http://localhost:8080/#/pup/player/",
grant_type: 'authorization_code'
},
json: true
};
var token;
var refresh_token;
await request.post(authOptions, function(error, response, body) { // also get the refresh token
if (!error && response.statusCode === 200) {
token = body.access_token;
refresh_token = body.refresh_token;
}
});
while (!token && !refresh_token){ // wait for both of them
await sleep(100)
}
this.token = token; // save them in the class instance properties
this.refresh_token = refresh_token;
}
Lastly the puppeteer fills in a html field with the token generated in step 6 on the SDK site and presses a button to start the SDK player.
// this function gets called after the button gets pressed
async function main(){
console.log(window.localStorage.getItem("__cid")) // print out all the data
console.log(window.localStorage.getItem("__csec"))
console.log(window.localStorage.getItem("__auth"))
console.log(getToken())
const player = new Spotify.Player({ // start the sporify player
name: 'Home Spotify Player',
getOAuthToken: cb => cb(getToken())
});
player.connect().then(()=>{ // connect the player
console.log(player)
});
window.player = player;
}
function getToken(){
return document.getElementById("token_input").value;
}
You are done. Next step for me at least was communicating using another UI page to the backend puppeteer to control the SDK page (play/pause/skip etc.) This process is pretty "hacky" and not pretty at all but if you just have a little personal project it should do the job fine.
If anyone would be interested in the whole code I might even upload it somewhere, but I think this read is long-enough and overly detailed anyway.
The proper way for this would be to use the device authorization grant flow - Spotify does this already for its TV applications, but they seem to block other applications from using it. It is possible to find clientIds online that are working with this, but it is not supported by Spotify.
I explained how this works and requested that they enable it in a supported way for custom applications in this feature request - please upvote the idea there if you find it useful.
That said, it is also possible to implement your own device authorization grant flow by hosting an extra server between your device and Spotify. That server should
host an authorize and a token API endpoint
host a user-facing page where the user can enter the user code
a callback page for Spotify to redirect the user after login
I believe this is how https://github.com/antscode/MacAuth implements it:
When the device calls the authorize, the server should generate a record containing the device_code and user_code and send them back in the response. The server should keep the record for later.
When the user enters the user_code in the user-facing page, the server should redirect the user to Spotify to login, and after login the user should be redirected to the server's callback page. At that moment the server can fetch credentials from Spotify's token endpoint using the data it received in the callback. The server should store the credentials it received in the record of the user_code.
The device can poll the server using the device_code for the availability of the tokens using the token endpoint.

Resources