I am trying to pull data from Hubspot to an Azure Data Factory environment by using the Hubspot Linked Service. I am at a stop due to an error, which I cannot resolve.
Steps I've followed:
Created a private app through this Hubspot guide
I got the right Client ID, Client Secret
I got the Access Token & Refresh Secret through Postmen
I made a Linked Service with this Microsoft guide
I get this
I am able to pull Contact info through Postmen.
I found these posts helpfull but they did not give me a proper answer:
Error when creating the linked service for Hubspot in ADF
https://www.loom.com/share/5880c34ba1a344998a019460fcbac5f6
https://community.hubspot.com/t5/APIs-Integrations/Connecting-HubSpot-from-Azure-Data-Factory/m-p/414894
Related
I am working on a project, in which I am building a publish-subscribe system through Azure Web PubSub Service. The JavaScript clients (subscribers) are connected through socket and able to receive the published message on the Hub and in the Group. For this I followed this tutorial: https://learn.microsoft.com/en-us/azure/azure-web-pubsub/tutorial-pub-sub-messages?tabs=javascript
Everything is working fine, as I am in the POC phase yet. So currently I am publishing messages through JavaScript from above tutorial. But now I am starting to integrate it in my existing app which is developed in PHP. So I am thinking to use following Azure REST API for Web PubSub operation: https://learn.microsoft.com/en-us/rest/api/webpubsub/dataplane/web-pub-sub
Before this I worked on the Azure Graph APIs, in which I did an App Registration at Azure Portal with some Redirect URLs, and assign some permissions to access the resources. I did this for Outlook Mail, Overdrive, SharePoint and its very straight forward process to get the access token (JWT) to access the resources.
But I am stuck and unable to get the access token for Azure Web PubSub API. I need token to just Publish a message from PHP end by REST API. I checked in App Registration's > API permissions section, there is no permission for the Azure Web PubSub Service. I checked the same app in Enterprise application also. Did some hit and try, but didn't get the access token. I know the issue is with the permission.
Please help me to get the token for this, so will call the Web PubSub API, Or if I am missing something please help me to trace. Any suggestion are welcome.
Thanks.
I think you can make use of the following REST API endpoints:
Web Pub Sub - Grant Permission
https://learn.microsoft.com/en-us/rest/api/webpubsub/dataplane/web-pub-sub/grant-permission#webpubsubpermission
Web Pub Sub - Generate Client Token
https://learn.microsoft.com/en-us/rest/api/webpubsub/dataplane/web-pub-sub/generate-client-token
Problem: I am trying to load data from Office 365 by using Azure Data Factory. When I test connection of the linked service I am getting error:
Office365 data loading failed to get OAuth token. Operation returned an invalid status code 'InternalServerError'
Screenshot of the error:
Tried: I have created linked service (Data store - Office 365). By following steps from this link: Microsoft - Getting started with Microsoft Graph Data Connect
I have registered Application via Azure AD-App Registration for generating Application (client) ID/Service principal ID and Client secrete/Service principle Key.
Two permission are there under API permission page in Application with Granted admin consent.
Please help me I am unable to find the reason , I can't find the missing gear to drive.
this is Guosen from ADF dev team.
Can you double check the service principal id and key? Making sure they are all correct.
I am trying to move files between Azure Data Factory and SharePoint Online.
I have some test data files in a SharePoint location and I have a pipeline in Data factory containing a Web Activity and a Copy data activity. The copy activity then links to a Data Lake as its sink.
My desired outcome is for the Copy Data activity to be granted access to the SharePoint Online API and then proceed to move the retrieved data into the data lake. As is stands it has the bearer token but cant access the data through the SharePoint Online API.
I have followed the Microsoft guide here, including the prerequisites.
I am getting the following error in Data Factory:
Error code 2200
"Failure happened on 'Sink' side. ErrorCode=UserErrorInvalidCredentialToReadHttpFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The credential to read http file is invalid.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (401) Unauthorized.,Source=System,
The error is present on the Copy data activity (Which uses a HTTP linked service), not the web activity. Therefore I am successfully getting a bearer token which is passed to the Copy data activity, only to be denied.
Troubleshooting:
I have assigned Owner role access to the data factory for the data
lake.
I have registered the service principal App as per the prerequisites
in the Microsoft guide above, successfully using the Application ID,
Key and Tenant Id to access the bearer token
I have granted SharePoint online full access to the registered
service app, successfully, as I can see it listed in the trusted apps in SharePoint. I did so not
only here:
https://my_company.sharepoint.com/_layouts/15/appinv.aspx but here
https://my_company.sharepoint.com/sites/folder/_layouts/15/appinv.aspx
I have granted full API permissions to SharePoint and Microsoft Graph
in the registered Service app.
I have replicated the process in Postman. Only to have the same
results - I retrieve a bearer token and then get 401 error when
trying calling a GET request for the files, using the bearer token.
The error in Postman is: {"error_description":"Exception of type 'Microsoft.IdentityModel.Tokens.AudienceUriValidationFailedException' was thrown."}
I 've read that an alternative OData linked service in Data Factory is not possible using using basic authentication here. So I have tried this article using AAD service principal with cert, but had further problems trying to generate a certificate. Perhaps this is my best bet.
Most relevant posts I've used:
How to move sharepoint list or excel file to azure sql dw?
https://learn.microsoft.com/en-us/answers/questions/109275/copy-file-from-sharepoint-not-working.html
http://www.ktskumar.com/2017/01/access-sharepoint-online-using-postman/
Azure Data Factory and SharePoint
So I found the solution to my question. Using an Azure Logic app to move the files from SharePoint Online to Azure storage.
This was infinitely easier than using data factory and I wish someone told me that a while ago.
Reference found here
Glad you found a workaround! In case this is still a relevant topic for you I found a solution.
A few weeks ago I ran into the same problem following the same documentations as you.
I was able to solve the problem by enabling Custom App Authentication. For SPO tenants that were created after 07.11.2018 the authentication via ACS app-only access token is disabled by default (see blue box in link below).
Granting access using SharePoint App-Only
I decided to write a blog post with a step by step guide on how to move multiple files from an SPO folder to Blob Storage using Azure Data Factory, maybe you will find it usefull:
Copy files from SharePoint to Blob Storage
The Setup :
Am trying to access Azure PowerBI based APIs using console application since my company wants to Suspend/Resume PBI capacity to optimize costing.
Have been successful in accessing those apis as per official documentation API Emulator Window, it works by perfect as it just asks with a authentication window for username and password of my outlook account and it generates authentication token implicitly successful.
When i tried to implement calling api from my console application which will be triggered automatically scheduled, there are ways to generate authentication token programmatically. Following are the methods i used to generate the same before accessing the PBI APIs.
Approach #1 : Generating token by using Azure Active Directory Authentication Libraries..
This approach asks to add AAD authentication libraries via Nuget, upon adding the same and it goes unsuccessful as the method AcquireTokenAsync takes no parameters but tutorial specified 4 parameters to be feed into this method (function overloading missing?). So i couldnt generate token using this approach. Surely a library version problem but the official MS documentation didnt explain anything above versions available over same.
Approach #2: Authorize Active Directory without dialog box StackOverflow Question with marked as answer
This approach upon calling HTTPResponseMessage, it gets hung up without going next line or catch statement.
Approach #3: Trying to emulate token generation using POSTMAN and using the generated token on my console app to check accessibility.
This approach gives a successful token generation but when using the generated token in the console app, it says unauthorised token.
Doubt Part:
Have generated ClientID, ClientSecretID and TenantID in Azure using AppRegisteration but dont know how this gets associated with PowerBI Service in azure. Do Azure Active Directory comes in place betweeen PBIService and AppRegisteration? Based on generated AppRegisteration details have tried to access this authentication api (https://login.microsoftonline.com/tenantId/oauth2/token) to generate token. This is successfully generates a token but miserably getting failed with unauthorised access on PBI api.
Am i badly missing something? will be helpful on what wrong about this concept of accessing PBI based API getting authenticated via console app. Asusual PBI community sites didnt help much.
1. How to manage Azure Power BI Embedded capacity
If you want to manage Azure Power BI Embedded capacity with rest API, please refer to the following steps.
create a service principal and assign Azure RABC role to the sp(I use Azure CLI)
az login
#it will create a service principal and assign contributor role to the sp
az ad sp create-for-rbac -n "jonsp2"
Get Token
Post https://login.microsoftonline.com/tenantId/oauth2/token
Content-Type: application/x-www-form-urlencoded
grant_type =client_credentials
&client_id=<sp app id>
&client_secret=<sp app password>
&scope=https://management.azure.com/
Call Rest API
POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.PowerBIDedicated/capacities/{dedicatedCapacityName}/suspend?api-version=2017-10-01
Authorization: Bearer <token>
2. How to call Power BI rest api
If you want to call Power BI rest api, please refer to the document and the document.
The detailed steps are as below
Register Azure AD application in Azure portal
Configure API permissions
Test (I test in postman)
a. get access token
b. call API
Suspend/Resume PowerBI API is described over here:
https://learn.microsoft.com/en-us/rest/api/power-bi-embedded/capacities/suspend
https://learn.microsoft.com/en-us/rest/api/power-bi-embedded/capacities/resume
Did you read the whole Azure REST API Reference?
All steps needed to send an HTTP request are documented over here:
https://learn.microsoft.com/en-us/rest/api/azure/
I've a requirement of refreshing the Power BI dataset from Azure data factory. I want to do it by calling the Power BI Rest API service. For authentication process I need a access token which I can get using a Client ID and secret. So, while registering the app I was wondering what type of redirect URI should I select? And how does it link with my task because I'm neither creating any web app or native app.
Regarding MS docs about creating Daemon application for authorization:
"During application registration, the reply URI isn't needed. You need to share secrets or certificates or signed assertions with Azure AD. You also need to request application permissions and grant admin consent to use those app permissions."
Please, read a whole documentation over here: https://learn.microsoft.com/en-gb/azure/active-directory/develop/scenario-daemon-overview