I have been trying to connect my SSIS package (on prem) to connect to my data lake store. I have installed the Azure Feature pack which has worked fine.
But when I create a Data Lake connection in my ssis package, I need the following .
Image of SSIS Azure Data Lake connector Manager
ADLS Host – which is fine I know how to get that.
Authentication ( Azure AD User Identity )
UserName & Password, - which I am having issues with.
My question is how do I define a username and password for my data lake?
You can find them in Azure AD User which is within the same subscription with your Azure Data Lake. Usually, it is your email address and password which you used to login Azure portal.
More details, you can refer to this documentation.
Related
I have setup a Synapse environment and filled my storage account with some sample Parquet files. I have then created a serverless SQL database and created some external tables over the Parquet files. All this works fine and I can query these tables fine from the Synapse UI and SSMS using AD Authentication.
The problem is I want to connect an app to the serverless SQL database which doesn't support AD authentication. Therefore I want to connect it using a standard SQL account. I have setup a SQL account (username and password) and I'm able to connect through SSMS, but not query any tables due to this error...
External table 'TableName' is not accessible because content of directory cannot be listed.
I assume this is a double-hop authentication problem because the SQL user doesn't have access to the storage account? I can't seem to find any guides on how to do this. Does anyone know?
I've written a blog-post where this issue is tackled, as I've encountered this problem as well a few days ago. You can read it here.
Basically, it comes down to the fact that you have to:
create a SQL login for your user
create a credential in SQL that has the same name as the URL that points to the container in your datalake that contains the files you want to query
grant reference rights on that credential to your SQL login
create a user on your database for that login
Next to that, you also need to create some specific role-assignments.
I am trying to create an ADF Linked Service connection to a Synapse Link Serverless SQL Pool connected to ADSL Storage. I can successfully get a connection but when I try and use a dataset to access the data I get a permission issue.
I can successfully access the data via Synapse studio :
This is the error I get when I use the data set in ADF.
I can also look at the schemas in SSMS , where they appear as External tables. But get a similar credential error at the same point.
Has anyone come across this issue please ?
There are a few pieces of information you haven’t supplied in your question but I believe I know what happened. The external table worked in Synapse Studio because you were connected to the Serverless SQL pool with your AAD account and it passed through your AAD credentials to the data lake and succeeded.
However when you setup the linked service to the Serverless SQL Pool Im guessing you used a SQL auth account for the credentials. With SQL auth it doesn’t know how to authenticate with the data lake so looked for a server scoped credential but couldn’t find one.
The same happened when you connected from SSMS with a SQL auth account I’m guessing.
You have several options. If it’s important to be able to access the external table with SQL auth you can execute the following to tell it how to access the data lake. This assumes the Synapse Workspace Managed Service Identity has Storage Blob Data Reader or Storage Blob Data Contributor role on the data lake.
CREATE CREDENTIAL [https://<YourDataLakeName>.dfs.core.windows.net]
WITH IDENTITY = 'Managed Identity';
Or you could change the authentication on the linked service to use the Managed Service Identity.
When I am looking at microsoft documentation of excels connections possibilities and it says that my Microsoft 365 Apps for enterprise can connect to Azure Blob Storage and Azure Data Lake Storage
I want to connect to my Datalake Gen 2 in excel, I'll go to the Data tab Get Data > From Azure > From Azure Blob Storage
Here is the question: How do I connect to my Gen 2 datalake with Azure AD / OAuth / username (user#domain.com) and password? Storage Account Key Access is disabled on this datalake. That includes Shared key and shared access signature (SAS)
I can see only a UI connector for ADLS Gen1
Alternately for ADLS Gen2....
Here is something I tried, you can use user delegation SAS Key, which is secured with Azure AD credentials.
Generate a user delegation SAS url for that blob you want to access.
Select Web Source to get Data,
If prompted, select Anonymous
First I want to thank KarthikBhyresh-MT for his input that inspired me to find the right solution.
The Solution
First I found the url to the desired file inside the datalake inside azure portal
I copied the changed the url and changed the word blob to dfs
In excel (office 365) I Get Data > From Web, and put in the altered url. From there I get this image, where I can log into the datalake with OAuth credentials
Then I could load the simple.csv file into excel, work on it and see the transactions in the datalake logs.
I am trying to copy data from Azure data lake Gen2 to Azure synapse(SQL data warehouse) through Azure data factory. Following are some details:
source(ADLS) linked service authentication type: service principal
sink(Synapse) linked service authentication type: managed identity
Copy method selected : Polybase
While validating, i am getting this error: "Source linked service should not have authentication method as Service principal".
when i selected "bulk insert" copy type, it works fine.. can anyone help me understand this? is it written anywhere that for polybase we should have same authentication type for linked service?
This is because direct copy by using PolyBase from Azure Data Lake Gen2 only support Account key authentication or managed identity authentication. You can refer to this documentation.
So if you want to direct copy by using PolyBase, you need change your authentication method to account key or managed identity.
There is a workaround, Staged copy by using PolyBase. You can refer to this documentation about this.
I created a scoped credential in a Azure SQL Datawarehouse database to create an external table over some files in a Azure Data Lake Store.
When I try creating the external table I get the message.
Msg 105061, Level 16, State 1, Line 35 Unable to find any valid
credential associated with the specified data source. Credential is
required to connect to Azure Data Lake Store.
How do I troubleshoot this? My AzureAD application has access to the storage. I use the same AD-application (with a different key) for my Azure Data Factory pipeline that stores the files in the Azure Data Lake Store.
I haven't found any commands that let you test your credentials and see what credentials the database tries to use or why it fails. Any ideas?
https://learn.microsoft.com/en-us/sql/t-sql/statements/create-database-scoped-credential-transact-sql
So I had missed adding my scoped credential when I created the external data source. So create the scoped credential first, then the external data source.