I'm trying to connect to Sharepoint Online (Sharepoint 365?) content using OAuth2 and the REST API. I need to do this from Python as it is an addition to an existing application. I have already managed to connect the application to Google Drive using OAuth2 and REST, so I think I understand the fundamentals of using OAuth2.
I've tried a number of combinations of places to configure the client_id and client_secret and authenticate and receive access and refresh tokens.
So far I have been able to receive a refresh token and use it to obtain an access token; however, I'm unable to use the access token to access content on the Sharepoint 365 site.
For configuring the client_id and client_secret I associated my Sharepoint site with AAD using the Azure Management Portal. Then I added an application to the Sharepoint AD entry with the client_id and client_secret. In Sharepoint I used appregnew.aspx to register the client_id and verified the application appears in appprincipals.aspx.
I call the authentication service using:
.../login.windows.net/common/oauth2/authorize?api-version=1.0&response_type=code&client_id=&redirect_uri=&resource=Microsoft.Sharepoint
and am able to authenticate, receive a code, call back to
.../login.windows.net/common/oauth2/token with the code and receive access and refresh tokens.
I serialized those tokens and from a separate process call
.../login.windows.net/common/oauth2/token with the refresh_token, client_id, client_secret, and grant_type=refresh_token and receive a new access token.
Finally I call in to Sharepoint service endpoint
-my.sharepoint.com/personal//_api/web/files'
with the new access token and it fails telling me the resource Microsoft.Sharepoint is invalid (Invalid audience Uri 'Microsoft.SharePoint')
I've been spinning my wheels trying various permutations of where the client_id is configured and this is as far as I've gotten. Since all the documentation and examples seem to depend on using C# libraries such as TokenHelper I feel like I'm missing something key but simple and can't find the required information.
Has anyone connected to Sharepoint Online using Python, Ruby, Java, etc? If so:
Where should the client_id be configured?
What are the endpoints for obtaining the refresh and access tokens?
What is the appropriate audience uri or resource for which to request tokens?
Many thanks!
I've successfully connected to SharePoint Online using PHP. It looks to me like you're combining two different methods to do so.
Here is what I did:
Register my app in Azure AD on the Management Portal to get client_id, client_secret, redirectUri and the right permissions on Office 365 for sites.
Call the authorization endpoint at https://login.microsoftonline.com/common/oauth2/authorize
Call the token service endpoint at https://login.microsoftonline.com/common/oauth2/token
Call the SharePoint REST endpoint with the access token.
The resource should be simply the URL of your SharePoint site.
The appregnew.aspx and appprincipals.aspx pages are used for apps for SharePoint but it doesn't look like you're building one.
I think you are trying to do OAuth from Sharepoint Online without creating an Add-In inside Sharepoint. This was the exact problem I was facing. I know this is an old question but my answer might be helpful to someone else coming here.
It is possible to do OAuth from any web-app. Here are the step by step instructions in my blog -
https://medium.com/#yash_agarwal2/performing-oauth-and-rest-calls-with-sharepoint-online-without-creating-an-add-in-677e15c8d6ab#.6pf4wp83b
High Level overview -
1) Get client id and secret by registering yourself here - https://sellerdashboard.microsoft.com/Registration
2) Get Access Token, Refresh Token by following steps here -
https://msdn.microsoft.com/en-us/library/office/jj687470.aspx
Im probably a bit late to the party, but to whoever stumbles into this one, you can have a look at how the SharePoint Oauth App Client lib does it.
You'll probably want to check the SPSite and SPAccessToken classes.
I know this is old by had to deal with similar problem and have 2.5 solutions.
This solution is specifically for the federated logins.
Simple hacky solutions works like a magic for files :)
On Windows 10 - the silliest solution:
First login to your sharepoint with Internet Exploer/Edge. Must be IE/Edge.
Then in Start-> Run dialog or any Windows Explorer address bar do this: \\<your_sharepoint_site.com>#SSL\rest\of_the\path. Note #SSL after host name.
Make sure to replace all the forward slashes with backslashes.
This makes sure that your login credentials are captured by the system.
Then navigate to "This PC", menu "Computer->Map Network Drive" and paste either that same path from above or your normal URL https://<your_sharepoint_site.com>/rest/of_the/path into Folder text field of Map Network Drive.
Now you have Sharepoint folder mapped as regular network drive X:\! Which allows you to treat is just as normal file system folder from any program or programming language.
This solution uses WebDav WebClient and by default you will hit a 50MB limit! Worry not follow these steps to release yourself from the shackles of the silly limit.
Simple Hacky.
Open your browsers and open the 'Developer Tools'.
Navigate to 'Network' tab of 'Developer Tools'
With Network tab open login to your sharepoint site.
Click on any of the requests and you shall see that every single Request Header has a Cookie with FedAuth=some_very_long_base64_encoded_xml_struct;rtFa=something_else_long.
Basically you need these two cookies FedAuth and rtFa.
Now with every GET POST (curl, wget etc) request to the Sharepoint just add a new header Cookie with the value FedAuth=<fedauth-cookie-value>;rtFa=<rtFa-cookie-value>.
Programmatic-hacky (example in python on windows), essentially #2 but cookie manipulation all done programmatically.
We shall steal the cookie directly from the kid... I mean from the Chrome browser.
Navigate to your Sharepoint With the chrome browser and perform all the necessary login actions.
Steal the cookie gist. Plagiarized from here.
Use the cookie gist.
REST of Sharepoint REST API wisdom here and ref here.
Finally open a beer enjoy your day.
Related
I'm asking this question on the back of a previous question I raised, as the scope of the question has changed somewhat but that might be worth reading first for background info.
I'm trying to programatically obtain data out of our Dynamics CRM instance, using a single set of admin credentials within a Node powered Express app. This Express app is hosted on a separate server outside of our network where CRM is hosted. The app will then request, process and serve CRM data back to any logged in user who has access (controlled by roles/permissions within the app), meaning an end user only has to login into the Express app.
From my web browser, if I visit our on-premise CRM endpoint: https://my.crm.endpoint, I get prompted for a username and password.
If I provide correct credentials, I am authenticated and have full access to the CRM, allowing me to query the API.
Example
https://my.crm.endpoint/api/data/v8.2/contacts?$select=fullname,contactid
This returns a lovely JSON object containing all the data I want :)
NOW! Under the covers, I can see that it is using NTLM to authenticate, of which I have little knowledge :/ Having read up a little and watched a few YouTube videos, I have a basic understanding of the challenge/response mechanism but I'm still unsure as to how to proceed.
NB: I have read this from Microsoft which describes the mechanism, but doesn't give any specific examples. I don't even know what hashing algorithm should be used, or what headers to set etc.
Question Can anyone provide any sort of detail as to how I can authenticate with our CRM using NTLM from a Web App (Express in my case)?
Steps I can see the browser making...
Visit https://my.crm.endpoint
302 Redirected to: https://fs.our.domain/adfs/ls/?wa=wsignin1.0&wtrealm=https%3a%2f%2fmy.crm.endpoint%2f&wctx=rm%3d1%26id%3dfaf0791c-6a3a-4c4e-9e69-9dfa8fd4c2e8%26ru%3d%252fdefault.aspx&wct=2018-04-20T10%3a12%3a37Z&wauth=urn%3afederation%3aauthentication%3awindows
Prompted for user credentials
** enter credentials**
A whole bunch of stuff happens here and I get a little lost but looks like it gets a couple of 401's then a POST is made to the https://my.crm.endpoint. Another 302 is shown, then finally a GET to the actual default.aspx page.
I then have access to CRM.
NB: Once authenticated, I can see three cookies that have been set and which are sent when querying the api example above. These cookies are MSISAuth, MSISAuth1 and ReClientId.
If I'm missing any crucial info, please let me know and I'll provide what I can!
UPDATE
I have just installed httpntlm module and attempted to authenticate using this...
let httpntlm = require('httpntlm');
httpntlm.get({
url: 'https://my.crm.endpoint',
username: '<my.email#address.com>',
password: '<mypassword>',
workstation: '', // unsure what to put here if anything?
domain: '' // unsure what to put here if anything?
}, function (err, res){
if(err) return err;
console.log(res.headers);
console.log(res.body);
});
The response I get is this...
{ location: 'https://fs.our.domain/adfs/ls/?wa=wsignin1.0&wtrealm=https%3a%2f%2fmy.crm.endpoint%2f&wctx=rm%3d1%26id%3d93a4c6fd-5b17-4a2b-965f-07af5e96b08f%26ru%3d%252fdefault.aspx&wct=2018-04-20T14%3a08%3a00Z&wauth=urn%3afederation%3aauthentication%3awindows',
server: 'Microsoft-IIS/8.5',
req_id: '298acefc-53aa-46fa-96c4-e5d8762b1fd2',
'x-powered-by': 'ASP.NET',
date: 'Fri, 20 Apr 2018 14:08:00 GMT',
connection: 'close',
'content-length': '397' }
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
Anyone able to shed any light on what I actually need to be doing?! :-/
UPDATE 2
Following #markgamache comment, and having read the suggested docs, we are indeed using WS-Fed! As the Wa=signin1.0 parameter informs the browser to pop up a login box, does this make it impossible to achieve this programmatically, without additional user interaction?
Based on my following understanding:
You are using CRM on premises with claims authentication (ADFS). This means that when an user accesses CRM, the user is redirected to ADFS for authentication (if the user is in the internal network, by default ADFS uses integrated Windows authentication) and then the user is redirected back to CRM.
You must call a CRM endpoint from an external (node.js) application. That call is NOT "client side" (i.e. via browser/javascript) but "server side" (i.e. from the web server hosting the application)
The ideal solution would be to apply here the S2S (server to server) scenario which involves an application user in CRM which in turn is used to call a CRM API using the OAuth client credentials flow (client id + secret). Problem is, as far as I know, currently the application user concept is only supported in CRM online, not on premises.
So then you can try one of these 3 options:
Despite the fact that you are using claims authentication in CRM, you can still use integrated Windows authentication (IWA). How? If you check the CRM IIS site, you must have an HTTPS binding. If you add an HTTP binding (i.e. port 80 no host header), you can access http://CRM_Server_Name/api/data/v8.2/contacts using IWA. So in this scenario, the httpntlm module you already tried could work. Please note that CRM supports one HTTPS IIS binding and one HTTP IIS binding - so make sure to not to add more of one binding of each type.
Mimic (programmatically of course) the authentication flow you observed in the browser. What does this mean? Generate an IWA authenticated request to https://fs.our.domain/adfs/ls/?wa=wsignin1.0&wtrealm=https%3a%2f%2fmy.crm.endpoint%2f&wauth=urn%3afederation%3aauthentication%3awindows. ADFS will authenticate you and will give you some cookies. You will need to store those cookies to make a subsequent request to https://my.crm.endpoint/api/data/v8.2/contacts. Not a great solution but should work.
Use OAuth. Problem is that as I described at the beginning, the ideal OAuth flow for this scenario (client credentials) is not available for CRM on prem as far as I know. So then you must use the Authorization Code Grant flow, described here. First you will need to register an ADFS application and then again, you will need to make several HTTPS calls (one of them will be a call to the ADFS IWA authentication endpoint) to finally obtain a token that you can use to make calls to the CRM endpoint.
that is a redirect to an ADFS server for claims based authentication. That page, or the next redirect, will ask you to authenticate with either cert, forms, or windows integrated (NTLM or Kerberos). If you pass auth, your browser will be given a token to send to https://my.crm.endpoint. The URL suggest the claims will be via WS*. https://blogs.technet.microsoft.com/askpfeplat/2014/11/02/adfs-deep-dive-comparing-ws-fed-saml-and-oauth/
Might seem a silly question, but Microsoft's documentation isn't very beginner friendly, I think. It uses as examples "http://localhost:31544" for the sign-on url and "http://MyFirstAADApp" for the redirect URI, but although I understand what a local host is I can't figure out what exactly the numbers on it are and how I define them for my application, and absolutely zero clue of what the redirect URI is supposed to do for a native application and how should I define a URI for my own.
To be more clear on what kind of app I'm trying to add, I merely want to acess the Office 365 management API tools and get some data from it, so I imagine a native app would fulfill my needs for now. Registering the app on Azure AD is required to do so according to Microsoft's documentation.
So expanding on the title, how to define an URI for my native app is what I would mainly like to know. Some further clarification on what exactly is the purpose of this URI as well as to how to use and/or define a localhost URL for an Web app would also be much appreciated.
I know this is ancient, but I don't see a satisfying answer here, and maybe someone will come across this and find it useful. To answer the question asked, unless you're going to work outside of the default MSAL handling of the server responses, and I don't expect you would from your description, I'd just go ahead and use the default:
https://login.microsoftonline.com/common/oauth2/nativeclient
When you go into the Azure AD portal, go to your application and, from the Overview, select the "Set RedirectURL" option, you'll add a platform and select the "Mobile and Desktop Applications" and you'll be provided with the choice of 3 URLs to choose from. My understanding is this is just there for custom handling of authorization tokens and is telling MS where to send those tokens. The MSAL library functions seem to use this link as well, so they're probably handling this in the backend.
I agree with the OP though, the MS docs are severely lacking for newcomers and I wasn't able to find an end-to-end description of what needs to happen to get, in my case, a desktop application to send email through Office365 using 2FA. I would forge ahead as best I could until I hit the next error, then explore that, sort it, then slam into the next one. Rinse and repeat. This was made extra tedious as I had to go through a 3rd party IT group to get the 2FA access codes every time I wanted to test.
Best of luck, hope this helps someone!
how to define an URI for my native app is what I would mainly like to
know.
You should provide a Redirect URI that is unique to your application as it will return to this URI when authentication is complete.
In your application, you will need to add a class level variables that are required for the authentication flow, include ClientId and Redirect URI.
Here is the diagram:
Native application makes a request to the authorization endpoint in Azure AD, this request includes the Application IP ,Redirect URI and application ID URI for the web api.
After user signed in, Azure AD issues an authorization code response back to the client application's redirect URI. After that, the client application stops browser interaction and extracts the authorization code from the response.
Then the client app use this code to sends a request to Azure AD's token endpoint. upon successful validation, Azure AD returns two tokens.
Over HTTPS, the client app uses the returned JWT access token to add the JWT string with a “Bearer” designation in the Authorization header of the request to the web API. The web API then validates the JWT token, and if validation is successful, returns the desired resource.
More information about it, please refer to this article.
For native you can set redirect to be equal to the Application ID URI, which now defaults to look like //app:{ApplicationId}
Redirect uri be starts with SSL URL, so select your project, enable SSL URL and use this auto generated SSL URL (for example : https://localhost:port#) as redirect uri , same to be updated in the azure app registration as additional redirect URIs
I have a web api, using azure ad oauth authentication. I want my associates´ web sites to be able to use this api to display content on their public website. To clearify: No user should be needing consenting/grant anything and they are anonymous.
I have used this guide to set up my web api and this code ("Web Application to Web API"->WebApp-WebAPI-OAuth2-AppIdentity-DotNet-code) for my demo client.
The default scope/permission was assigned to client app in azure portal, like so
The actual problem I trying to solve is my different associates webpages should have different permissions. I was hoping to create different scopes and assign different associates client apps with different scopes and I could check if they for example has update_customer_x-scope.
All this "seems" to work ok. I get my authentication token and I am able to get content from the web api. BUT I dont get any scopes at all. Reading the first guide(web api) I see that you should be able to get the default scope by using the code:
if (ClaimsPrincipal.Current.FindFirst("http://schemas.microsoft.com/identity/claims/scope").Value == "user_impersonation")
{/*you have the default scope*/}
But I dont find anything here, and no scopes at all. Any clues?
This is the content of the JWT-token, as requested
The TodoListController.cs from WebApp-WebAPI-OAuth2-AppIdentity-DotNet just use the client credentials flow as astaykov commented for retrieving the access token without user interaction, at this time, if you decode your code via https://jwt.io/, you would not find the scp property.
For your requirement, you need to follow WebApp-WebAPI-OpenIDConnect-DotNet for calling a web API with the signed-in user's permissions.
I want to trigger a custom action when a document/file gets added on Sharepoint Online. The idea was to use Microsoft Flow, but as Flow doesn't allow you to work with the custom columns of the file added (just the title, date added, content type, etc), I was thinking of calling the Sharepoint API by hand, via an Azure Function. So the steps would be:
A user adds a new document on a document library on Sharepoint Online
A Microsoft Flow gets triggered after that document gets added.
The Flow calls an Azure Function and pass the document id to it.
The function authenticates to Microsoft Graph Api, and then request for the file with that id.
The function get the custom fields and then it does something with the custom parameters of that file.
However, I ve been trying a lot to authenticate, but it always require me to put user and pass on an HTML. Of course, the Azure Function wouldn't be able to do that. How can I authenticate to Microsoft Graph API easily? I'm stuck with that.
Any other alternative? As I said before, this is a workaround I ve been thinking because Microsoft Flow doesn't allow you to get the custom fields of a recently added/modified file, just the default ones... But what I just want to do is to manage those data when a new file is added. Solution doesn't necessarily need to use a Function or Microsoft Graph Api.
Thanks in advance.
There's not a great way to do the authentication right now - we're working on support for this. The hard part is the AD authentication, which any solution will need to deal with. We're doing some things to make this easy with Functions.
What you can do for auth is get offline access that grants you access to a refresh token, and then use the refresh token to get your access token. You can One way to build this is on Azure Website's existing EasyAuth token store (https://cgillum.tech/2016/03/07/app-service-token-store/). Then you can log in once, save the refresh token in EasyAuth's token store, and the function can pull it from there per execution.
That's the high level answer. I can elaborate on more specific steps if you're eager to do this now.
You could register the function or de Remote Web (Azure or on premise) as an app in your Sharepoint, via URL:
"http://.sharepoint.com/_layouts/15/AppRegNew.aspx
And give it permissions via:
http://.sharepoint.com/_layouts/15/AppInv.aspx
For example:
<AppPermissionRequest Scope="http://sharepoint/content/sitecollection" Right="FullControl" />
</AppPermissionRequests>
In this way you will be capable of accessing your tenant or list or whatever your permissions are.
What you are seeking is app-only authentication. If you can make your document queries/updates via the SharePoint Rest API, then you have 2 options:
Granting access via Azure AD App-Only
Granting access using SharePoint App-Only (This is the same method indicated in #netadictos' answer above.)
If you prefer to make the document queries/updates with the MS Graph API, then only #1 is an option. #1 is a bit more involved, because it requires a client secret AND a self-signed security certificate. #2 will only require the app/client ID and secret.
The MSDN documentation linked above uses a PowerShell script to generate the security cert, but I prefer Bob German's instructions for manually creating/exporting one. He also includes instructions for registering an Azure AD application for your Azure function in his tutorial.
I have the following scenario:
1.- A web api project in Azure, that I want to secure using Azure AD (I don't mind Token, cookie, whatever, as far as it meets the entire scenario)
2.- An Azure web site in asp.net MVC, also secured by Azure AD. This website has to call the web api controller with SSO (I'm using same Azure AD in the entire scenario)
3.- Some JavaScript code running in a page in SharePoint Online, also calling the web api controller in any secure way (The Office 365 tenant is also using same Azure AD). If you don't know about SharePoint, let's say I have an SPA project where I can only use Javascript and html (no server side code).
Following some of the MS Azure AD samples and some blogs from Vittorio Bertocci I'm able to get the points 1 and 2 working fine, using OWIN and Oppen ID connect. However, seems impossible to achieve point 3. As I'm inside a page in SharePoint Online, I can only use javascript, and not any server side code. I'd like to get a valid token for the current user, that is already logged in SP, and remember that SP uses same Azure AD that web api site.
Can I call the Azure AD and get a valid token, just from client code?
I'm open to any possible solution. I can do whatever in the web api project. If you are thinking in a SharePoint app with an appPart, and the appPart calls the web api from server side code, I agree that will work, but it's an option that is not allowed at the moment :(
Many thanks.
I have similar needs. While waiting for a Microsoft sponsored solution we’re working on the following approach.
3) in Your solution (i.e. HTML page with JavaScript, hosted in SharePoint Online and running in Browser) will call Services in 1) (i.e. Web Api Service layer in Azure).
In Our case we only want to validate that the calls made from SharePoint Online (via users browser, i.e. JavaScript) originate from a correct Office 365 / SharePoint Online user in our tenant.
We are opting out of using the App Model as we only want some simple HTML / JavaScript pages in our Intranet and don’t want App Webs. The Web Api server side code is kind of our “Web Part” code behind.
Change to the solution after trying it out and having workable code:
The auth cookies are ReadOnly and cannot be used. Instead we have registered one metod in our service layer as App in SharePoint Online (via appregnew.aspx). That methods url (e.g. https://cloudservice.customer.com/api/authentication/token) is registered as App start page in the app manifest and is deployed to a site Collection.
Now we can call our App via https://customer.sharepoint.com/sites/devassets/_layouts/15/appredirect.aspx?instance_id={GUID} i a jQuery ajax call and parse the result. AppRedirect sends the correct SPAuthToken which we use in our service endpoint (i.e. App start page) to call back to SharePoint and check context.Web.CurrentUser. User email is then stored in Table Storage with a generated Token which we send back to the caller (i.e. the jQuery ajax call to app redirect).
That token is then used in all other service layer calls in order to be sure of who is calling our service layer and in some cases perform authorization in our service layer.
Note, You can use the same approach in order to store Refresh and AccessToken in your client and provide that in all calls to your service from your client and use those tokens in order to do App Calls back to SharePoint. This enables HTML UI in SharePoint host webs and server code using user context in Azure service layer.
To follow up, ADAL.js has recently been released, and the ability to use CORS with O365 APIs was recently added, enabling a scenario for script clients to communicate with services protected by Azure AD, such as your Web API.
http://www.andrewconnell.com/blog/adal-js-cors-with-o365-apis-files-sharepoint
UPDATE 2018:
This is now supported by SharePoint Online and the SPFx development model, and officially documented, for instance here
Consume enterprise APIs secured with Azure AD in SharePoint Framework
Being said that the work done meanwhile by Vittorio, Kirk, and their teams, but extending that also to Andrew that has delivered great samples, is awesome; that doesn't really fully reply the original question because one of the requirements is to don't run the component as Add-in Part.
If you try to use ADAL JS (which starts its own OAuth flow) hosting that directly in a SP page, that's not going to work, or anyway you can expect a weird behavior for the user (cause of client redirects happening on the browser).
The solution proposed by Peter Karpinski is interesting, and will work matching the requirements in the original question, but requires quite some complexity and additional management/resources.
This recent article provides an alternative solution similar to Peter's one, but requiring less 'extras' and somewhat simpler, also reusing user's SP identity.
Consuming Azure Hosted Web API from SharePoint Online using JavaScript and Office 365 identities
and doesn't either require the use of ADAL on the client side and the implementation of custom security provider / token issuer on the server side.
The identity (cookie) will be passed via properly handling CORS (documentation) on both sides.
However, as you can read in my comments to that blog, this won't work normally with IE due to its security zone implementation. You'll have to be sure you have control on IE security zones on the clients, or have an alternative solution specific for IE.
As of today AAD does not support the OAuth2 implicit flow (or OpenId Connect variants) hence you can't obtain a token from AAD using a user-agent (browser), regardless of whether you hit the wire handcrafting the protocol or using a library.
However keep an eye on future announcements, as this is an important scenario for us!
HTH
Cheers,
V.
update we now support the implicit flow on our server, and we released a library for helping you consume the new feature: http://www.cloudidentity.com/blog/2015/02/19/introducing-adal-js-v1/
Thank youi for r your patience!
The fact that you say you can use only HTML/JS let me guess you're having a SharePoint-hosted App.
Azure AD Authentication Library (ADAL) doesn't provide yet in this moment support for HTML5.
I've been looking since a while (almost one year) on how to implement something as you say; but I couldn't find any working solution, which doesn't make use also of some 'code-behind'.
I'd suggest you then to move to a Provider-hosted App, where you'll be able to have some C# code, and make use of ADAL to retrieve and reuse the OAuth token.
Also, I think is worth to look at this documentation page:
Choose patterns for developing and hosting your app for SharePoint
at section Match your hosting pattern with your development goals
thanks for your help.
Well, it's not a SP-Hosted App, but it's same scenario. It's just a SP page in SP Online, so I can only use JS code like in a SP-hosted app.
As I said in my question, I agree the Provider hosted app is likely the right (or at least, the unique) solution, but that means to build and app, deploy it, and add teh appPart manually to the page (is not easy to package in a WSP). So, this solution is quite hard, when you only want to make some AJAX calls and show some data.
However, after all that I've seen, I think we can't do anything else. I'm gonna wait some more days to see if someone know any weird workarround that could work, and if not, I'll mark your answer as valid.
Thanks again!