We deploy out our infrastructure using ARM and configure the Git link as such:
Firstly a variable is configured:
"repoConfiguration": {
"type": "FactoryVSTSConfiguration",
"accountName": "[parameters('accountName')]",
"repositoryName": "[parameters('repositoryName')]",
"collaborationBranch": "[parameters('collaborationBranch')]",
"rootFolder": "[parameters('rootFolder')]",
"tenantId": "[parameters('repositoryTenantId')]",
"projectName": "[parameters('projectName')]"
},
and then that variable is used when provisioning the resource:
"type": "Microsoft.DataFactory/factories",
"name": "[variables('dataFactoryName')]",
"apiVersion": "2018-06-01",
"location": "[resourceGroup().location]",
"properties": {
"repoConfiguration": "[if(equals(parameters('environmentShort'), 'dev'), variables('repoConfiguration'), json('null'))]",
"publicNetworkAccess": "Disabled"
},
"identity": {
"type": "SystemAssigned"
},
Note that we are setting the Tenant ID for the tenant where Git resides.
Let's just say our emails for Git are:
joebloggs#company.com
and for Azure (the other tenant):
joebloggs#xxcompany.com
The problem is, the behaviour doesn't match this article from Microsoft.
The behaviour is as follows:
First we log on using our joesbloggs#xxcompany.com email and verify via MFA. The we choose the appropriate subscription and Data Factory. Once that is logged in, we are presented with the following pop up which doesn't 100% match what is in the Appendix of the article:
From here, we get the popup to "Pick an account" and we select the joebloggs#company.com email and go through the usual MFA.
Once ADF loads up, we then have this issue where (and my assumption) it isn't handshaking with the two logins correctly:
When I click on anything in ADF e.g. the toolbox, ADF reverts to this:
Checking the repo setup, it is setup:
I'm at a loss on what could be the issue. Even when manually configuring Git, it works whilst the browser is open, but closing the browser and opening again yields the same issue.
We are also using a Self Hosted IR for ADF.
For anyone who encounters this issue, the cure is:
Open Browser
In the first tab, open Azure Devops and log in
Open Azure Portal and log in using the other tenant account in the 2nd tab
Open adf.azure.com in the 3rd tab. Log in using your "other tenant account" and when it tries to connect to git, it will ask you to select the desired account. Select the account that you have used for Azure Devops and cross tenancy should work fine between Data Factory and Azure Git
Related
I was asked by Azure Support to post this question, just to see if anyone had a useful opinion.
I am stepping through MS Azure training courses. I created the usual free account to go through these. I've gone through a few dozen of them, and am now at this one:
https://learn.microsoft.com/en-us/learn/modules/secure-and-isolate-with-nsg-and-service-endpoints/3-exercise-network-security-groups?source=learn
This attempts to use the Azure PowerShell service. I had some trouble getting to the PowerShell page. It appears that if I'm not already logged into the portal, it goes into a semi-infinite loop, trying to get to the shell page, then trying to login, then the shell page, and finally it gives up and says "We couldn't sign you in. Please try again.".
However, I was able to work around this. If in a separate tab, I log into the Azure Portal, and then go back and follow the link to Azure Cloud Shell, it passes the login gate and sends me to the page where I choose Bash or PowerShell. The course specifies using Bash. When I select that, it then asks me to create a Storage object. When I confirm that, it gives me the following error (subscription id elided):
{
"error": {
"code": "RequestDisallowedByPolicy",
"target": "cs733f82532facdx4f04x95b",
"message": "Resource 'cs733f82532facdx4f04x95b' was disallowed by policy. Policy identifiers: '[{\"policyAssignment\":{\"name\":\"Enforce tag on resource\",\"id\":\"/subscriptions/xxxxx/providers/Microsoft.Authorization/policyAssignments/740514d625684aad84ef8ca0\"},\"policyDefinition\":{\"name\":\"Enforce tag on resource\",\"id\":\"/subscriptions/xxxxx/providers/Microsoft.Authorization/policyDefinitions/be3862a6-ca1e-40b0-a024-0c0c7d1e8b3e\"}}]'.",
"additionalInfo": [
{
"type": "PolicyViolation",
"info": {
"policyDefinitionDisplayName": "Enforce tag on resource",
"evaluationDetails": {
"evaluatedExpressions": [
{
"result": "True",
"expressionKind": "Field",
"expression": "tags[Department]",
"path": "tags[Department]",
"targetValue": "false",
"operator": "Exists"
}
]
},
"policyDefinitionId": "/subscriptions/xxxxx/providers/Microsoft.Authorization/policyDefinitions/be3862a6-ca1e-40b0-a024-0c0c7d1e8b3e",
"policyDefinitionName": "be3862a6-ca1e-40b0-a024-0c0c7d1e8b3e",
"policyDefinitionEffect": "deny",
"policyAssignmentId": "/subscriptions/xxxxx/providers/Microsoft.Authorization/policyAssignments/740514d625684aad84ef8ca0",
"policyAssignmentName": "740514d625684aad84ef8ca0",
"policyAssignmentDisplayName": "Enforce tag on resource",
"policyAssignmentScope": "/subscriptions/xxxxx",
"policyAssignmentParameters": {
"tagName": {
"value": "Department"
}
}
}
}
]
}
}
I think the simple conclusion from this is that my free account doesn't have enough rights to do what is needed here. The documentation I've read seems to imply that I have to get additional rights on the account in order to do this. However, I'm just using a free account that I created to go through the Azure training courses. It doesn't really make sense to ask me to do this. I've seen other Azure courses create a temporary sandbox supposedly because they have particular objects pre-created in the sandbox, but I'm also thinking that the sandbox has particular permissions that are not available in the free account. It seems to me that the only reasonable fix for this problem is for that course to be refactored to use a temporary sandbox with the correct permissions.
I'm just looking for any opinions on this, and confirmations that this is what should be done.
It doesn't look like you are creating resource, cloudshell storage, on your free subscription. Except if you added to a Work/Corporate tenant.
From the information you provide, subscription you are trying to use has a policy to enforce tags Department, mean any resource created should have a tag with Department information.
I have a requirement, where I needed to fetch the tenantDirectoryId for a given subscription.
I could find a rest get api https://management.azure.com/subscriptions/[subscription]/versions...
The error response to this gave the tenant directory.
Is there any better way to fetch directory for a subscriptionId.
As of now (06/07/2018), an easy approach would be running az account show in the Azure Cloud Shell (requires a Storage Account) in the Azure Portal.
--- Command ---
az account show
--- Command Output ---
{
"environmentName": "AzureCloud",
"id": "{Subscription Id (GUID)}",
"isDefault": true,
"name": "{Subscription Name}",
"state": "Enabled",
"tenantId": "{Tenant Id (GUID)}",
"user": {
"cloudShellID": true,
"name": "{User email}",
"type": "user"
}
}
See this doc for more details on Azure Cloud Shell: https://learn.microsoft.com/en-us/azure/cloud-shell/overview
We have recently had the same problem we wanted to fix. After speaking to Microsoft there is no supported way to do this through the Graph API when accessing as an application using the client_credentialflow.
An approach we have taken to get the tenant id is to call the OAuth2 metadata document endpoint for your tenant
https://login.microsoftonline.com/{tenant}/v2.0/.well-known/openid-configuration
By calling this you’ll see your token endpoint as well as other useful information listed which you can utilise to log in. Or you can grab the tenant id using string stripping.
I want to set up CI/CD (ARM template) with StreamAnalytics Job with output set to DataLake Store.
https://learn.microsoft.com/en-us/azure/templates/microsoft.streamanalytics/streamingjobs/outputs#microsoftdatalakeaccounts
The issue comes with refreshToken:
"It is recommended to put a dummy string value here when creating the data source
and then going to the Azure Portal to authenticate the data source
which will update this property with a valid refresh token"
Furthermore after 90-days refresh token is outdated and you need to do "Renvew Authorization"
https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-data-lake-output#renew-data-lake-store-authorization
I tried to authorize ServicePrincipal.
How to do automatic deployment for ASA with DataLake?
How to handle issue with this 90-days token validitiy?
Maybe you wiped the trail :)
at this time it is not yet possible. Sorry for the inconvenience. However we know this is very important and we will add Service Principal auth in the near future (we are looking at the exact ETA).
In the meantime you need to renew manually the token. This can be done without losing any data by (1) stopping the job, (2) changing the token, then (3) restarting the job last time it was stopped.
Let me know if you have any further question.
As far as I know quite soon MSI-based authentication will be in preview.
But if you need an immediate solution (to e.g. be able to have a VSTS pipeline running through without error) you can do the following:
Create template (e.g. with the CICD NuGet Package [1])
Manipulate the ARM Template <jobName>.JobTemplate.json
Add the output datasource object for the ADLS output object
If you work with Visual Studio you can get the values quite easily from the ADLS output JSON
It is important to set refreshToken to some fake value
Like the following:
"outputs": [
{
"name": "xxx",
"properties": {
"serialization": {
"type": "Json",
"properties": {
"encoding": "UTF8",
"format": "LineSeparated"
}
},
"datasource": {
"type": "Microsoft.DataLake/Accounts",
"properties": {
"accountName": "xxx",
"tenantId": "xxx-xxx-xxx-xxx-xxx",
"tokenUserPrincipalName": "xxx#xxx.com",
"tokenUserDisplayName": "xxx, xxx",
"filePathPrefix": "xxx/{date}/{time}",
"dateFormat": "yyyy/MM/dd",
"timeFormat": "HH",
"refreshToken": "faketoken"
}
}
}
},
...
Deploy the ARM Template
The job will start successfully but it is necessary to renew the token, therefore
Stop the job
Renew the authentication of the ADLS output
Start the job
Resources
[1] CICD NuGet Package
I am using a Classic Web Service with a non-default endpoint for a Update Resource activity on the Azure Data Factory. This is the error I get:
Screenshot of Error
I didn't find any info on the web and couldn't figure it out myself. This website shows an example that I used by just filling in my values for mlEndpoint, apiKey and updateRessourceEndpoint:
{
"name": "updatableScoringEndpoint2",
"properties": {
"type": "AzureML",
"typeProperties": {
"mlEndpoint": "https://ussouthcentral.services.azureml.net/workspaces/xxx/services/--scoring experiment--/jobs",
"apiKey": "endpoint2Key",
"updateResourceEndpoint": "https://management.azureml.net/workspaces/xxx/webservices/--scoring experiment--/endpoints/endpoint2"
}
}
}
There is no mention of a token that needs to be passed...
this error is basically saying the apiKey you provided is invalid to perform the update resource operation. Here is some posts for your reference: https://social.msdn.microsoft.com/Forums/azure/en-US/3bb77e37-8860-43c6-bcaa-d6ebd70617b8/retrain-predictive-web-service-programmatically-when-do-not-have-access-to-managementazuremlnet?forum=MachineLearning
Please also be noted that if you modified your linked service in ADF, remember to re-deploy the pipeline as well to reflect your change in time.
As of yesterday I added a webapi and a native app to my Azure AD.
Yesterday I was able to download the WEBAPI's manifest file to add appPermissions (for delegating permissions to the native app).
Not there's no longer any appPermissions located in the manifest file, it has changed. I have tried to add new apps, and even a new AD.
Now theres only appRoles there.
Does anyone know why this is, and how I could solve this so i still can delegate permissions?
Might this be the new thing to use?
oauth2Permissions": [
{
"adminConsentDescription": "Allow the application full access to the service on behalf of the signed-in user",
"adminConsentDisplayName": "Have full access to the service",
"id": "23906603-82b7-4471-afbd-b6bcf2fe7b1a",
"isEnabled": true,
"origin": "Application",
"type": "User",
"userConsentDescription": "Allow the application full access to the service on your behalf",
"userConsentDisplayName": "Have full access to the service",
"value": "user_impersonation"
}
],
#Simon Agren is right, and that new oauth2Permissions worked for me. Just download an old existing Maninifest and you will see how it now uses the oauth2Permissions instead of the appPermissions. Remember to create a new GUID for the id attribute.
I also turned surprised when I saw that this morning :)
Cheers.