I want to use the invoke-restmethod in a foreach loop to upload a multiplefiles through Infile parameter - azure

Each time the loop executes the file is being replaced in the place of the first file . I want to upload it as a new file without distrubing the existing ones..
foreach ($blob in $blobs)
{
$file=New-TemporaryFile
$file=Get-AzureStorageBlobContent -Container $container_name -Blob $blob.Name -Context $ctx -Destination $localFile -Force
$contents = Get-Content $localFile -Raw -ErrorAction:SilentlyContinue
$f=New-TemporaryFile
Add-Content $f $contents
$Header = #{
"Content-Disposition"="attachment;filename=$($blob.Name)"
"Authorization"=$accessToken
}
Invoke-RestMethod -Uri $apiUrl -Headers $Header -Method put -InFile $f
}

I think you over-did it with the New-TemporaryFile commands in your code.
Why use Get-AzureStorageBlobContent to store blob content in a file and then create another temp file to copy the contents to?
This would be much simpler:
foreach ($blob in $blobs) {
$localFile = New-TemporaryFile
Get-AzureStorageBlobContent -Container $container_name -Blob $blob.Name -Context $ctx -Destination $localFile.FullName -Force
$Header = #{
"Content-Disposition"="attachment;filename=$($blob.Name)"
"Authorization"=$accessToken
}
Invoke-RestMethod -Uri $apiUrl -Headers $Header -Method put -InFile $localFile.FullName
}
Edit
The New-TemporaryFile command returns a System.IO.FileInfo object which you cannot use directly as the InFile parameter. Instead, it expects a full path and file name, so we use $localFile.FullName

Related

powershell upload blob to azure blob storage set content type

I am making a ReST api call and uploading the JSON payload to azure blob storage using the below powershell.
PoSh:
$Params = #{"URI" = 'https://myapiexample.com/api/data/'}
$Result = Invoke-RestMethod #Params | ConvertTo-Json
$context=New-AzStorageContext -StorageAccountName "mystorage" -StorageAccountKey ""
$container=Get-AzStorageContainer -Name "input" -Context $context
$content = [system.Text.Encoding]::UTF8.GetBytes($Result)
$blobpath = "azblob/mydata.json"
$container.CloudBlobContainer.GetBlockBlobReference($blobpath).UploadFromByteArray($content,0,$content.Length)
$container.CloudBlobContainer.GetBlockBlobReference($blobpath).Properties.ContentType = "application/json"
$container.CloudBlobContainer.GetBlockBlobReference($blobpath).SetProperties()
I notice that when the blob is stored in azure blob storage the content-type is application/octet-stream whereas I would want it to be application/json. The code I use is not working as expected.
After reproducing from my end, I could able to achieve your requirement using Set-AzStorageBlobContent. Below is the complete code that is working for me that sets the content type of the blob.
$url = "<YOUR_URL>"
$dest = "samplex.json"
Invoke-WebRequest -Uri $url -OutFile $dest
$context = New-AzStorageContext -StorageAccountName "<YOUR_ACCOUNT_NAME>" -StorageAccountKey "<YOUR_ACCOUNT_KEY>"
Set-AzStorageBlobContent -Context $context -Container "containers" -Blob "mydata.json" -File $dest -Properties #{"ContentType" = "application/json"};
RESULTS:

Undeleting a Soft Deleted Blob in Azure Storage Using a REST API call from PowerShell

I am trying to create a script to retrieve blobs for a given customer number from a storage account in Azure. All blobs reside in a single container, with 'actioned' blobs being soft deleted.
I can use PowerShell to display the relevant blobs, including their 'IsDeleted' status, but I understand that PowerShell doesn't have the necessary command to undelete blobs and so I'm trying to make a REST API call from the PowerShell script.
I do an inital login to the Azure platform and set a variable for an SAS token (which includes the necessary permissions to undelete):
$username = "<myUserName>"
$encryptedPwd = Get-Content <path\securepassword.txt> | ConvertTo-SecureString
$cred = New-Object System.Management.Automation.PsCredential($username, $encryptedPwd)
$strgaccname = "<myStorageAccount>"
$strgcontainer = "<myContainer>"
#SAS Token
$sastkn = "<mySAStoken>"
#Set StorageContext
$ctx = New-AzStorageContext -StorageAccountName $strgaccname -SasToken $sastkn
$subId = "mySubscriptionID"
Connect-AzAccount -Credential $cred -Subscription $subID
I can list all matching blobs with the following PowerShell:
$searchstring = '*'+<myCustomerNumber>+'*'
Get-AzStorageBlob -Blob $searchstring -Context $ctx -Container $strgcontainer -IncludeDeleted `
| Select-Object Name, Length, LastModified, IsDeleted `
| Sort-Object LastModified -Descending
I am unsure how to proceed with the REST API call. Looking at some other people's methods, I have something like the following, using a test blob that has been soft deleted:
$uri = "https://<myStorageAccount>.blob.core.windows.net/<myContainer>/<myTestBlob>?comp=undelete"
$headers = #{
'Authorization' = "Bearer <accessToken>";
'x-ms-date' = $((get-date -format r).ToString());
'x-ms-version' = "2020-12-06";
}
Invoke-RestMethod -Method 'Put' -Uri $uri -Headers $headers
However, I don't know how to create the Bearer Access Token that is mentioned.
We have done a repro in our local environment & it is working fine, Below statements are based on our analysis.
You can use the below Powershell script which will help you in restoring the soft-deleted blobs in your storage account.
Here is the Powershell Script :
Connect-AzAccount
#Get all deleted blob within a container
$StorageAccount = Get-AzStorageAccount | Where-Object { $_.StorageAccountName -eq "<storageAccountName>" }
$Blobs = Get-AzStorageContainer -Name "<ContainerName>" -Context $StorageAccount.Context | Get-AzStorageBlob -IncludeDeleted
$DeletedBlobs=$($Blobs| Where-Object {$_.IsDeleted -eq $true})
#Get your Bearer access token
$resource = “https://storage.azure.com"
$context = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile.DefaultContext
$accessToken = [Microsoft.Azure.Commands.Common.Authentication.AzureSession]::Instance.AuthenticationFactory.Authenticate($context.Account, $context.Environment, $context.Tenant.Id.ToString(), $null, [Microsoft.Azure.Commands.Common.Authentication.ShowDialog]::Never, $null, $resource).AccessToken
#Restore
foreach ($DeletedBlob in $DeletedBlobs) {
Write-Host "Restoring : $($DeletedBlob.Name)"
$uri = "$($DeletedBlob.BlobBaseClient.Uri.AbsoluteUri)?comp=undelete"
$headers = #{
'Authorization' = "Bearer $accessToken";
'x-ms-date' = $((get-date -format r).ToString());
'x-ms-version' = "2020-12-06";
}
Invoke-RestMethod -Method 'Put' -Uri $uri -Headers $headers
}
Here is the Sample output for your reference:
Note:
In order to perform the restoration of soft-deleted blob, you need to have a Storage Blob Data Contributor RBAC role on the Storage Account.

How to write data in Azure Blob storage programmatically?

I am using below PowerShell script to read JSON data using REST API call from source. Now I want to load the data of $Result to the Azure Blob Storage. Any idea please?
$Params = #{
"URI" = 'https://3ea5e53b-817e-4c41-ae0b-c5afc1610f4e-bluemix.cloudant.com/test/_all_docs?include_docs=true'
}
$Result = Invoke-RestMethod #Params | ConvertTo-Json -Depth 9
Regarding the issue, you can use the following ways
Save the JSON into one file then upload the file to Azure blob
$Params = #{
"URI" = 'https://3ea5e53b-817e-4c41-ae0b-c5afc1610f4e-bluemix.cloudant.com/test/_all_docs?include_docs=true'
}
$Result = Invoke-RestMethod #Params | ConvertTo-Json -Depth 9
$Result | Out-File "D:\file.json"
$context=New-AzStorageContext -StorageAccountName "andyprivate" -StorageAccountKey ""
Set-AzStorageBlobContent -File "D:\file.json" `
-Container "" `
-Blob "file.json" `
-Context $context `
-StandardBlobTier Hot
Directly upload to Azure blob
$Params = #{
"URI" = 'https://3ea5e53b-817e-4c41-ae0b-c5afc1610f4e-bluemix.cloudant.com/test/_all_docs?include_docs=true'
}
$Result = Invoke-RestMethod #Params | ConvertTo-Json -Depth 9
Write-Host "the result is :"
$Result
$context=New-AzStorageContext -StorageAccountName "andyprivate" -StorageAccountKey ""
$container=Get-AzStorageContainer -Name "input" -Context $context
$content = [system.Text.Encoding]::UTF8.GetBytes($Result)
$container.CloudBlobContainer.GetBlockBlobReference("my.json").UploadFromByteArray($content,0,$content.Length)

Use Powershell to Publish to a WebApp Virtual Directory

I have an Azure WebApp, that is split into two virtual directories - UI and API.
I've managed to create the virtual directories in code, but cannot find a means of publishing to them.
Here's my code so far:
# Set UI Virtaul Directory (call /ui )
$website = Get-AzWebApp -Name $appsvWebAppName -ResourceGroupName $resourceGroupName
$VDApp = New-Object Microsoft.Azure.Management.WebSites.Models.VirtualApplication
$VDApp.VirtualPath = "/ui"
$VDApp.PhysicalPath = "site\wwwroot\ui"
$VDApp.PreloadEnabled ="YES"
$website.siteconfig.VirtualApplications.Add($VDApp)
$website | Set-AzWebApp -Verbose
# Set API Virtual Directory (call /api )
$website = Get-AzWebApp -Name $appsvWebAppName -ResourceGroupName $resourceGroupName
$VDApp = New-Object Microsoft.Azure.Management.WebSites.Models.VirtualApplication
$VDApp.VirtualPath = "/api"
$VDApp.PhysicalPath = "site\wwwroot\api"
$VDApp.PreloadEnabled ="YES"
$website.siteconfig.VirtualApplications.Add($VDApp)
$website | Set-AzWebApp -Verbose
$website.SiteConfig.VirtualApplications
# Dotnet publish & convert to zip here, removed for brevity ...
$uiZipPath = $zipFilesFolder + "\ui.zip"
$publishprofile = Get-AzWebAppPublishingProfile -ResourceGroupName $resourceGroupName `
-Name $appsvWebAppName `
-OutputFile $publishProfileFileName
Publish-AzWebApp -ArchivePath $uiZipPath `
-ResourceGroupName $resourceGroupName `
-Name $appsvWebAppName
I can't see how to point Publish-AzWebApp at a virtual directory.
The publish can be done manually, but I really want to automate it (using Publish-AzWebApp or another means).
How can I do this please?
The Publish-AzWebApp does not support that, you could use Kudu API in powershell to automate it.
In my sample, it uses VFS to create the directory first, then upload the zip file via Zip.
$appsvWebAppName = "xxxxxxx"
$resourceGroupName = "xxxxxxx"
$resource = Invoke-AzResourceAction -ResourceGroupName $resourceGroupName -ResourceType Microsoft.Web/sites/config -ResourceName "$appsvWebAppName/publishingcredentials" -Action list -ApiVersion 2018-02-01 -Force
$username = $resource.Properties.publishingUserName
$password = $resource.Properties.publishingPassword
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $username, $password)))
$userAgent = "powershell/1.0"
# Create the folder, not lose `/` after `ui`
$apiUrl = "https://$appsvWebAppName.scm.azurewebsites.net/api/vfs/site/wwwroot/ui/"
Invoke-RestMethod -Uri $apiUrl -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)} -UserAgent $userAgent -Method PUT
#Upload the zip file
$apiUrl = "https://$appsvWebAppName.scm.azurewebsites.net/api/zip/site/wwwroot/ui"
$filePath = "C:\Users\joyw\Desktop\testdep.zip"
Invoke-RestMethod -Uri $apiUrl -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)} -UserAgent $userAgent -Method PUT -InFile $filePath -ContentType "multipart/form-data"
For the site\wwwroot\api, it is the same logic, just change ui to api in the script.

Can you download an Azure Blob using Virtual Machine Identity Access Token through PowerShell without using Access Keys

I'm setting up a VM to do some bootstrapping on creation.
Part of this is to download a blob from an azure storage account to the VM.
These are all in the same subscription, resource group, etc.
I can do it this way fine:
function Get-BlobUsingVMIdentity
{
param(
[Parameter(Mandatory = $true)] $containerName,
[Parameter(Mandatory = $true)] $blobName,
[Parameter(Mandatory = $true)] $outputFolder
)
write-host "Defining package information"
mkdir $outputFolder -force
write-host "Getting Instance meta data"
$instanceInfo = Invoke-WebRequest -UseBasicParsing -Uri 'http://169.254.169.254/metadata/instance/?api-version=2018-02-01' `
-Headers #{Metadata="true"} `
| select -expand content `
| convertfrom-json `
| select -expand compute
$storageAccountName = "$($instanceInfo.resourceGroupName.replace('-rg',''))sa" # This is custom since we know our naming schema
$resourceGroupName = $($instanceInfo.resourceGroupName)
$subscriptionId = $($instanceInfo.subscriptionId)
write-host "Got storageAccountName [$storageAccountName], resourceGroupName [$resourceGroupName], subscriptionId [$subscriptionId]"
write-host "Getting VM Instance Access Token"
$response = Invoke-WebRequest -UseBasicParsing -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' `
-Headers #{Metadata = "true" }
$content = $response.Content | ConvertFrom-Json
$access_token = $content.access_token
write-host "Getting SAS Token From Storage Account"
$params = #{canonicalizedResource = "/blob/$($storageAccountName)/$($containerName)"; signedResource = "c"; signedPermission = "rcw"; signedProtocol = "https"; signedExpiry = "2031-09-23T00:00:00Z" }
$jsonParams = $params | ConvertTo-Json
$sasResponse = Invoke-WebRequest -UseBasicParsing -Uri "https://management.azure.com/subscriptions/$($subscriptionId)/resourceGroups/$($resourceGroupName)/providers/Microsoft.Storage/storageAccounts/$($storageAccountName)/listServiceSas/?api-version=2017-06-01" `
-Method POST `
-Body $jsonParams `
-Headers #{Authorization="Bearer $access_token"}
$sasContent = $sasResponse.Content | ConvertFrom-Json
$sasCred = $sasContent.serviceSasToken
write-host "Manually download blob"
$params = #{signedResource = "c"; signedPermission = "rcw"; signedProtocol = "https"; signedExpiry = "2031-09-23T00:00:00Z" }
$jsonParams = $params | ConvertTo-Json
$sasResponse = Invoke-WebRequest -UseBasicParsing -Uri "https://$($storageAccountName).blob.core.windows.net/$($containerName)/$($blobName)?api-version=2017-06-01" `
-Method POST `
-Body $jsonParams `
-Headers #{Authorization="Bearer $access_token"}
$sasContent = $sasResponse.Content | ConvertFrom-Json
$sasCred = $sasContent.serviceSasToken
write-host "Setting up storage context"
$ctx = New-AzStorageContext -StorageAccountName $storageAccountName -SasToken $sasCred
write-host "Downloading package"
Get-AzStorageBlobContent `
-Blob $blobName `
-Container $containerName `
-Destination $outputFolder `
-Context $ctx `
-Force
}
This works fine, except I have to grant full/write access to the identity in order for it to use the access key.
Is there a similar approach that would allow read only access to the blob?
My goals are:
1. No credentials stored anywhere
2. Download blob to VM from azure storage
3. No statically defined variables (ex: subscriptionid)
4. Read only access to the blob/storage account.
Appreciate any help!
According to my understanding, you want to use Azure VM MSI to access Azure storage. If so, please refer to the following steps:
Enable a system-assigned managed identity on a VM
Connect-AzAccount
$vm = Get-AzVM -ResourceGroupName myResourceGroup -Name myVM
Update-AzVM -ResourceGroupName myResourceGroup -VM $vm -AssignIdentity:$SystemAssigned
Grant your VM access to an Azure Storage container
Connect-AzAccount
$spID = (Get-AzVM -ResourceGroupName myRG -Name myVM).identity.principalid
New-AzRoleAssignment -ObjectId $spID -RoleDefinitionName "Storage Blob Data Reader" -Scope "/subscriptions/<mySubscriptionID>/resourceGroups/<myResourceGroup>/providers/Microsoft.Storage/storageAccounts/<myStorageAcct>/blobServices/default/containers/<container-name>"
access blob
# get AD access token
$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' `
-Headers #{Metadata="true"}
$content =$response.Content | ConvertFrom-Json
$access_token = $content.access_token
# call Azure blob rest api
$url="https://<myaccount>.blob.core.windows.net/<mycontainer>/<myblob>"
$RequestHeader = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$RequestHeader.Add("Authorization", "Bearer $access_token")
$RequestHeader.Add("x-ms-version", "2019-02-02")
$result = Invoke-WebRequest -Uri $url -Headers $RequestHeader
$result.content
Update
According to my test, when we get token to access Azure blob, we need to change resouce as https://storage.azure.com/
# get AD access token
$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://storage.azure.com/' `
-Headers #{Metadata="true"}
$content =$response.Content | ConvertFrom-Json
$access_token = $content.access_token
# call Azure blob rest api
$url="https://<myaccount>.blob.core.windows.net/<mycontainer>/<myblob>"
$RequestHeader = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$RequestHeader.Add("Authorization", "Bearer $access_token")
$RequestHeader.Add("x-ms-version", "2019-02-02")
$result = Invoke-WebRequest -Uri $url -Headers $RequestHeader
$result.content
Update
Found a much easier way to do it!
The key was to use:
Connect-AzAccount -identity
This auto logs in a the identity and allows you to interact through AZ module instead of trying to hack API calls together.
You only need [Storage Blob Data Reader] assigned to the container to pull this off.
Working example looks like this:
function Get-BlobUsingVMIdentity
{
param(
[Parameter(Mandatory = $true)] $containerName,
[Parameter(Mandatory = $true)] $blobName,
[Parameter(Mandatory = $true)] $outputFolder
)
write-host "Get instance info"
$instanceInfo = Invoke-WebRequest -UseBasicParsing -Uri 'http://169.254.169.254/metadata/instance/?api-version=2018-02-01' `
-Headers #{Metadata="true"} `
| select -expand content `
| convertfrom-json `
| select -expand compute
$storageAccountName = "$($instanceInfo.resourceGroupName.replace('-rg',''))sa" # This is custom since we know our naming schema
write-host "Clear existing identies to keep cache fresh"
Clear-AzContext -force
write-host "Authenticate using identity"
$account = Connect-AzAccount -identity
if(-not $account.Context.Subscription.Id)
{
write-error "Failed to authenticate with identity. Ensure VM has identity enabled and is assigned the correct IAM roles"
return
}
write-host "Get storage context"
$ctx = New-AZStorageContext -StorageAccountName $storageAccountName
write-host "Getting blob"
Get-AzStorageBlobContent `
-Blob $blobName `
-Container $containerName `
-Destination $outputFolder `
-Context $ctx `
-Force
}
Get-BlobUsingVMIdentity `
-containerName "deploy" `
-blobName "deploy.zip" `
-outputFolder "c:\deploy\"

Resources