Azure Powershell Function App Reading Content From Data Lake Gen 1 - azure

I would be very grateful for any assistance with this problem.
All I am trying to do is use a simple Azure function app using Powershell to read a file ("/input/specimen.json") in from my data lake, work with it, then eventually write it back out to the lake. I'm stuck already.
Here is the relevant code from the function app:
# The below line is at the very top of the function app run.ps1
using namespace System.Management.Automation.PSCredential
# The lines below are in the main area of the function app which are called.
$path = "/input/specimen.json"
$account = "mydatalake"
$tenant = "<my-tenant-id>"
$user = "<application/client id>"
$pass = "<client secret>"
$password = ConvertTo-SecureString $pass -AsPlainText -Force
$cred = New-Object -TypeName System.Management.Automation.PSCredential -argumentlist $user,$password
connect-azaccount -credential $cred -tenant $tenant -ServicePrincipal
$data = Get-AzDataLakeStoreItemContent -accountname $account -path $path
write-host $data
$body = $data | out-string
disconnect-azaccount
The output from the function app is:
2020-06-04T19:49:09Z [Error] ERROR: Get-AzDataLakeStoreItemContent
: Error opening a Read Stream for file /stan/interests.json.
Operation: GETFILESTATUS failed with HttpStatus:Forbidden
RemoteException: AccessControlException GETFILESTATUS failed with
error 0x83090aa2 (Forbidden. ACL verification failed. Either the
resource does not exist or the user is not authorized to perform the
requested operation.). [a6cfab98-149a-46eb-8cb6-bd0a40a11796] failed
with error 0x83090aa2 (Forbidden. ACL verification failed. Either the
resource does not exist or the user is not authorized to perform the
requested operation.).
[a6cfab98-149a-46eb-8cb6-bd0a40a11796][2020-06-04T12:49:08.1314705-07:00]
JavaClassName: org.apache.hadoop.security.AccessControlException.
The error reeks of ACLS and permissions, but I have set up a Service Principal to use for the 'connect-azaccount' and have made that SP both a Contributor and Owner of the data lake, but still get the same errors. I'm reasonably certain that the SP I am using IS being used to authenticate because the Function App logs show the created session being echoed out to the host with the correct tenant and client id.
This is beginning to drive me crazy. What else do I need to get this to work? Could there be other roles or permissions needed? Is there a better way to do this?
Any and all assistance would be greatly appreciated - thanks in advance.

If you want to use the service principal to access Azure data lake gen1 file, we need to configure ACL fro the sp. For more details, please refer to the document.
The ACL for Azure data lake gen1 has three permissions. There are Read, Write, and Execute. For more details, please refer to the docuemnt.
For example, we want to read file /Oregon/Portland/Data.txt, we need to configure ACL as below
Regarding how to configure it, we can do that on Azure Portal

Related

Automate connecting to Azure for checking and creating resources without prompt

I am required to write a PowerShell script, which can connect to my company's Azure account, check and create Azure resources (eg. Service Bus namespace, Service Bus topic, and Service Bus subscriptions). Everything worked well until I tried to deploy my script as a step in my project's on-premise TeamCity. I keep getting this error message
Exception calling "ShouldContinue" with "2" argument(s): "Windows
PowerShell is in NonInteractive mode. Read and Prompt functionality is
not available."
I investigated and found out that the problem is in this line
Connect-AzAccount
If I run the script manually, it will pop up a prompt asking me to login to Azure. I believe that's what went wrong. Because my project's on-premise TeamCity does not seem have an option to open a prompt for PowerShell command. I have read some workarounds, even on this website, but none of them is applicable to my case. Even a solution like https://stackoverflow.com/a/61099568/8213536 gave me these errors
WARNING: Unable to acquire token for tenant 'organizations' with error
'UsernamePasswordCredential authentication failed: There was an error
parsing WS-Trust response from the endpoint. This may occur if there
is an issue with your ADFS configuration. See
https://aka.ms/msal-net-iwa-troubleshooting for more details. Error
Message: Object reference not set to an instance of an object. See the
troubleshooting guide for more information.
https://aka.ms/azsdk/net/identity/usernamepasswordcredential/troubleshoot'
Connect-AzAccount : UsernamePasswordCredential authentication failed:
There was an error parsing WS-Trust response from the endpoint. This
may occur if there is an issue with your ADFS configuration. See
https://aka.ms/msal-net-iwa-troubleshooting for more details. Error
Message: Object reference not set to an instance of an object. See the
troubleshooting guide for more information.
https://aka.ms/azsdk/net/identity/usernamepasswordcredential/troubleshoot
At line:1 char:1
Connect-AzAccount -Credential $creds
+ CategoryInfo : CloseError: (:) [Connect-AzAccount], AuthenticationFailedException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand
One of the other solutions https://stackoverflow.com/a/52014189/8213536 requires an application's principal id, which is not applicable for my scenario either, as I am not creating a new application. I just need to be able to automatically connect to Azure (without prompt), check and create SB Namespace, SB Topic and SB Subscription.
Could someone please help me on this? Thanks.
As promised, I would like to post my solution. First I created a service principal with a client secret key. Then I asked my company's cloud engineer to assign it to the Azure subscription of my company and to the resource group that I intended to group all my necessary resources into. Finally in my code, I implemented something similar to https://stackoverflow.com/a/61099568/8213536
$applicationId = $azServicePrincipalId
Write-Host "Connecting to Azure using principal $applicationId"
$securePassword = $azServicePrincipalPw | ConvertTo-SecureString -AsPlainText -Force
$credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $applicationId, $securePassword
Connect-AzAccount -ServicePrincipal -TenantId $azureTenantId -Credential $credential
$azServicePrincipalId and $azServicePrincipalPw came from the Service Principal itself, while $azureTenantId came from my company's Azure subscription.
It is now working as expected.

Azure Function Connect-PnpOnline using Password for app

I created a powershell script to connect to my sharepoint site online.
Having enabled two-factor authentication, I set a password for the app for authentication and used it in the Connect-PnpOnline command.
$securePassword = ConvertTo-SecureString "myappapassword" -AsPlainText -Force
$credentials = New-Object PSCredential ("my#username.com", $securePassword)
Connect-PnPOnline -Url $TenantSiteURL$SiteRelativeURL -Credentials $credentials
On the local computer I don't get any errors, but when I try to run it in Azure Function I get:
Error validating credentials due to invalid username or password
Below are the few workaround to solve the above issue:
Please make sure that you have uploaded your pnp powershell module to Azure function .
By Navigating to Azure function >Advance tool>Kudu console> wwwroot folder > Create new folder copy and paste your modules that have in your local.
Add your service account user name and password in configuration of Azure function.
Try to replace the following cmd in your cmdlt:
$Credential = New-Object System.Management.Automation.PSCredential($serviceAccountEmail, $SecurePassword)
To use managed identity you can refer this Blog
For more information please refer the below links :
MS DOC: Connect-PnpOnine, Granting access via Azure AD App-Only
MS Q&A: Connecting with PnP PowerShell(without username and password)

Programmatically authenticate into AAD with MFA via powershell

I have a PowerShell script that logs into Azure subscription with the command Connect-AzAccount using user's credentials.
The code is the following:
$userPassword='password'
$userName="username"
$tenantId="########-####-####-####-############"
$subscriptionId="########-####-####-####-############"
$azureSecpassword = $userPassword | ConvertTo-SecureString -asPlainText -Force
$azureCredential = New-Object System.Management.Automation.PSCredential($userName, $azureSecpassword)
Connect-AzAccount -Credential $azureCredential -Tenant $tenantId -SubscriptionId $subscriptionId
The code above works without any user interaction.
Few days ago the customer enabled the multi-factor authentication for the users.
How can I keep a fully automated login process (without user interactions) with the multi-factor authentication?
Best Regards.
This is a common question. Unfortunately, the answer is No. If the account is MFA-enabled, you could just login with an interactive way.
In such a case, we choose to use the service principal to login with non-interactive in general.
$azureAplicationId ="Azure AD Application Id"
$azureTenantId= "Your Tenant Id"
$azurePassword = ConvertTo-SecureString "client secret" -AsPlainText -Force
$psCred = New-Object System.Management.Automation.PSCredential($azureAplicationId , $azurePassword)
Connect-AzAccount -Credential $psCred -TenantId $azureTenantId -ServicePrincipal
Reference - Sign in with a service principal.
If you must log in as a user, there might be 2 optional approaches.
1. If you will run the script locally or in a specific PC
You can Persist Azure user credentials. You can enable auto save, or manually save the context to a file, and then use it in another PS session.
If you enabled auto save, then you can directly get the context as following:
Get-AzContext
# If you have more than one contexts, you can choose one by specifing the name
Get-AzContext -Name 'CSP Azure (e5b0****-****-****-****-5e5f****4c68) - jack#h****a.onmicrosoft.com'
If you want to manually do it, here is the sample:
# Interactively log for one time
Connect-AzAccount
# Save the context
Save-AzContext -Path D:\ctx.dat
And in another PS session, you can:
Import-AzContext -Path D:\ctx.dat
2. Use refresh token to acquire token, and connect to Azure
You can get the refresh token from the auto saved Azure context (usually at C:\Users\<UserName>\.Azure\TokenCache.dat).
Open the dat file with notepad, and you will get the refresh token:
Then you can get a new token in PowerShell with that refresh token, and connect to Azure:
Clear-AzContext
$tenantId = "e4c9ab4e-****-****-****-230b****57fb"
$subscriptionId = "e5b0fcfa-****-****-****-5e5f****4c68"
$refreshToken = 'AQABAAAAAAAP0****a lot of characters here*****0A9FWoB8mvDtoWRJHBVO7GJzodLKYmNIAA'
$url = "https://login.microsoftonline.com/" + $tenantId + "/oauth2/token"
$body = "grant_type=refresh_token&refresh_token=" + $refreshToken
$response = Invoke-RestMethod $url -Method POST -Body $body
$AccessToken = $response.access_token
Connect-AzAccount -AccountId "the user id, jack#h****a.onmicrosoft.com" -AccessToken $AccessToken -Tenant $tenantId -SubscriptionId $subscriptionId
How can I keep a fully automated login process (without user interactions) with the multi-factor authentication?
You can't do this with a user account--that's the whole point of multi-factor authentication.
Instead, Azure AD supports authenticating with a service principal (instead of a user principal, like you're doing currently), and Azure supports granting access to Azure resources to service principals.
MFA requirements (and other conditional access policies) do not apply to service principals (often referred to as an Azure AD "app"), and service principals support more secure methods of authentication for automation scenarios (e.g. public/private key pairs).
So, what you should do:
Ensure the machine running this script is secure. Anyone with access to the machine has the same amount of access as the script.
Create an application identity and associate credentials with it.
Note: It is strongly recommend you use certificate-based authentication for your service principal, instead of password-based. It is a very insecure practice to have any kind of secret stored in a PowerShell script!
Grant the service principal the minimum level of access to Azure resources, to allow it to complete the required task.
Update your script to use the app's identity (service principal) instead of the user's identity. It's even simpler than using a user account:
$tenantId = "########-####-####-####-############"
$subscriptionId = "########-####-####-####-############"
$appId = "########-####-####-####-############"
$thumbprint= "##############"
Connect-AzAccount -ServicePrincipal -TenantId $tenantId -ApplicationId $appId -CertificateThumbprint $thumbprint
Note: If this script is running on a VM in Azure, you should forget step 2, and simply enable a managed identity and use that.

Applying PnP Provisioning template with Azure Functions/Powershell

I'm currently trying to develop a Flow which creates a SharePoint site and then utilizes an Azure Function to provision said site with a selection of libraries and such.
The Flow itself is pretty straight forward, but the bit I'm struggling with is the Azure Function itself, particularly the authentication; It uses a Connect-PnPOnline cmdlet, and with our tenant using MFA I've had a bit of difficulty getting around it - I've created a Service Principal User which at the very least has enabled me to connect to the site, but I still can't actually use Apply-PnPProvisioningTemplate, nor any other PnP commands really - when I try and do so, I get:
"The remote server returned an error: (403) Forbidden."
My Service Principal user has Contribute permissions, and a scope of Sites.FullControl.All, so I'm not sure where I'm going wrong.
See below the Powershell code - I don't think the issue is really arising from there but it can't hurt to show:
$requestBody = Get-Content $req -Raw | ConvertFrom-Json
"destination": "SharePointSiteURL"
$destination = $requestBody.destination
$secpasswd = ConvertTo-SecureString $env:pass -AsPlainText -Force
$mycreds = New-Object System.Management.Automation.PSCredential ($env:user, $secpasswd)
Connect-PnPOnline -url $destination -AppID $env:user -AppSecret $env:pass
Apply-PnPProvisioningTemplate -path "D:\home\site\wwwroot\template.xml"
Out-File -Encoding Ascii -FilePath $res -inputObject "Done $destination"
I should note that this is actually my first time using Azure Functions so forgive me if the solution here is quite simple! Any help would be appreciated.
Referring to MSDN Thread
https://social.msdn.microsoft.com/Forums/en-US/53629265-47da-41f7-b780-35eaba999f73/applying-pnp-provisioning-template-with-azure-functionspowershell?forum=AzureFunctions
This post work https://www.sharepointfire.com/2018/04/sharepoint-online-pnp-site-provisioning-using-flow-and-azure-function/

Running New-AzureRmResourceGroupDeployment from within a Function App

I need to wire up a stateless worker ad-hoc to perform a long running job based off a user action that self destructs when its done. I am trying to run New-AzureRmResourceGroupDeployment from within a PoSh Function App and cannot figure out how to authenticate to Azure from within the PoSh script.
I tried this:
$accountName = "myID#mydomain.com"
$pwd = ConvertTo-SecureString "password" -AsPlainText -Force
$cred = new-object PSCredential($accountName, $pwd)
Add-AzureRmAccount -Credential $cred
New-AzureResourceGroupDeployment -ResourceGroupName yadda yadda
And I get an error message that I need to use an Organization ID (which I am, our Azure AD is federated and we use AD Sync (and SiteMinder w/o WS-* if that matters)):
Add-AzureRmAccount : -Credential parameter can only be used with Organization ID credentials. For more information, please refer to http://go.microsoft.com/fwlink/?linkid=331007&clcid=0x409 for more information about the difference between an organizational account and a Microsoft account.
I tried "Login-AzureRMAccount -Credential $cred" with similar results.
If I do the Add- or Login- cmdlets from a PoSh window on my local machine (which is member joined to AD) with the -Credential flag I get a similar error. If I run the cmdlets without the credential I am prompted for credentials through an interactive ID/PW window (I do not have to enter my password once I type in my ID).
Does anyone know how I can do the authentication? I would be okay with authenticating like above, some sort of pass through credential from our web layer, or even an Option C I don't know about.
You will need to use service principal for authentication. A sample with instructions can be found here.
Azure Function role like permissions to Stop Azure Virtual Machines
For that you would need to use Service Principal auth. I don't think there is any sense of copypasting Azure Doc's to this answer, just consult this document:
https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-group-authenticate-service-principal

Resources