Undeleting a Soft Deleted Blob in Azure Storage Using a REST API call from PowerShell - azure

I am trying to create a script to retrieve blobs for a given customer number from a storage account in Azure. All blobs reside in a single container, with 'actioned' blobs being soft deleted.
I can use PowerShell to display the relevant blobs, including their 'IsDeleted' status, but I understand that PowerShell doesn't have the necessary command to undelete blobs and so I'm trying to make a REST API call from the PowerShell script.
I do an inital login to the Azure platform and set a variable for an SAS token (which includes the necessary permissions to undelete):
$username = "<myUserName>"
$encryptedPwd = Get-Content <path\securepassword.txt> | ConvertTo-SecureString
$cred = New-Object System.Management.Automation.PsCredential($username, $encryptedPwd)
$strgaccname = "<myStorageAccount>"
$strgcontainer = "<myContainer>"
#SAS Token
$sastkn = "<mySAStoken>"
#Set StorageContext
$ctx = New-AzStorageContext -StorageAccountName $strgaccname -SasToken $sastkn
$subId = "mySubscriptionID"
Connect-AzAccount -Credential $cred -Subscription $subID
I can list all matching blobs with the following PowerShell:
$searchstring = '*'+<myCustomerNumber>+'*'
Get-AzStorageBlob -Blob $searchstring -Context $ctx -Container $strgcontainer -IncludeDeleted `
| Select-Object Name, Length, LastModified, IsDeleted `
| Sort-Object LastModified -Descending
I am unsure how to proceed with the REST API call. Looking at some other people's methods, I have something like the following, using a test blob that has been soft deleted:
$uri = "https://<myStorageAccount>.blob.core.windows.net/<myContainer>/<myTestBlob>?comp=undelete"
$headers = #{
'Authorization' = "Bearer <accessToken>";
'x-ms-date' = $((get-date -format r).ToString());
'x-ms-version' = "2020-12-06";
}
Invoke-RestMethod -Method 'Put' -Uri $uri -Headers $headers
However, I don't know how to create the Bearer Access Token that is mentioned.

We have done a repro in our local environment & it is working fine, Below statements are based on our analysis.
You can use the below Powershell script which will help you in restoring the soft-deleted blobs in your storage account.
Here is the Powershell Script :
Connect-AzAccount
#Get all deleted blob within a container
$StorageAccount = Get-AzStorageAccount | Where-Object { $_.StorageAccountName -eq "<storageAccountName>" }
$Blobs = Get-AzStorageContainer -Name "<ContainerName>" -Context $StorageAccount.Context | Get-AzStorageBlob -IncludeDeleted
$DeletedBlobs=$($Blobs| Where-Object {$_.IsDeleted -eq $true})
#Get your Bearer access token
$resource = “https://storage.azure.com"
$context = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile.DefaultContext
$accessToken = [Microsoft.Azure.Commands.Common.Authentication.AzureSession]::Instance.AuthenticationFactory.Authenticate($context.Account, $context.Environment, $context.Tenant.Id.ToString(), $null, [Microsoft.Azure.Commands.Common.Authentication.ShowDialog]::Never, $null, $resource).AccessToken
#Restore
foreach ($DeletedBlob in $DeletedBlobs) {
Write-Host "Restoring : $($DeletedBlob.Name)"
$uri = "$($DeletedBlob.BlobBaseClient.Uri.AbsoluteUri)?comp=undelete"
$headers = #{
'Authorization' = "Bearer $accessToken";
'x-ms-date' = $((get-date -format r).ToString());
'x-ms-version' = "2020-12-06";
}
Invoke-RestMethod -Method 'Put' -Uri $uri -Headers $headers
}
Here is the Sample output for your reference:
Note:
In order to perform the restoration of soft-deleted blob, you need to have a Storage Blob Data Contributor RBAC role on the Storage Account.

Related

powershell upload blob to azure blob storage set content type

I am making a ReST api call and uploading the JSON payload to azure blob storage using the below powershell.
PoSh:
$Params = #{"URI" = 'https://myapiexample.com/api/data/'}
$Result = Invoke-RestMethod #Params | ConvertTo-Json
$context=New-AzStorageContext -StorageAccountName "mystorage" -StorageAccountKey ""
$container=Get-AzStorageContainer -Name "input" -Context $context
$content = [system.Text.Encoding]::UTF8.GetBytes($Result)
$blobpath = "azblob/mydata.json"
$container.CloudBlobContainer.GetBlockBlobReference($blobpath).UploadFromByteArray($content,0,$content.Length)
$container.CloudBlobContainer.GetBlockBlobReference($blobpath).Properties.ContentType = "application/json"
$container.CloudBlobContainer.GetBlockBlobReference($blobpath).SetProperties()
I notice that when the blob is stored in azure blob storage the content-type is application/octet-stream whereas I would want it to be application/json. The code I use is not working as expected.
After reproducing from my end, I could able to achieve your requirement using Set-AzStorageBlobContent. Below is the complete code that is working for me that sets the content type of the blob.
$url = "<YOUR_URL>"
$dest = "samplex.json"
Invoke-WebRequest -Uri $url -OutFile $dest
$context = New-AzStorageContext -StorageAccountName "<YOUR_ACCOUNT_NAME>" -StorageAccountKey "<YOUR_ACCOUNT_KEY>"
Set-AzStorageBlobContent -Context $context -Container "containers" -Blob "mydata.json" -File $dest -Properties #{"ContentType" = "application/json"};
RESULTS:

how do you use rest to put a .csv file onto a storage account?

how do you use resenter code heret to upload data to a storage account in the format of .csv
Get token
Get Request using invoke-rest method.
export data in csv to a storage account
$Request = Invoke-RestMethod #ParamRequest
Request.value.properties | export-csv -path $path -NoTypeInformation
API: https://learn.microsoft.com/en-us/rest/api/consumption/usage-details/list
Not really a PowerShell expert :) but essentially the idea is to create a Shared Access Signature (SAS) URL for the blob and then use that SAS URL for uploading the content directly into Azure Storage.
Here's what I came up with:
$accountName = "storage-account-name"
$accountKey = "storage-account-key"
$containerName = "blob-container-name"
$blobName = "blob-name.csv"
# Get storage context
$context = New-AzStorageContext -StorageAccountName $accountName -StorageAccountKey $accountKey
# Get Shared Access Signature (SAS) Token expiration time. I have set it to expire after 1 hour.
$sasExpiry = (Get-Date).AddHours(1).ToUniversalTime()
# Get a SAS Token with "write" permission that will expire after one hour.
$sasToken = New-AzStorageBlobSASToken -Context $context -Container $containerName -Blob $blobName -Permission "w" -ExpiryTime $sasExpiry
# Create a SAS URL
$sasUrl = "https://$accountName.blob.core.windows.net/$containerName/$blobName$sasToken"
# Set request headers
$headers = #{"x-ms-blob-type":"BlockBlob"}
# Set request content (body)
$body = "This is the content I wish to upload"
#Invoke "Put Blob" REST API
Invoke-RestMethod -Method "PUT" -Uri $sasUrl -Body $body -Headers $headers -Content-Type "text/csv"

How to write data in Azure Blob storage programmatically?

I am using below PowerShell script to read JSON data using REST API call from source. Now I want to load the data of $Result to the Azure Blob Storage. Any idea please?
$Params = #{
"URI" = 'https://3ea5e53b-817e-4c41-ae0b-c5afc1610f4e-bluemix.cloudant.com/test/_all_docs?include_docs=true'
}
$Result = Invoke-RestMethod #Params | ConvertTo-Json -Depth 9
Regarding the issue, you can use the following ways
Save the JSON into one file then upload the file to Azure blob
$Params = #{
"URI" = 'https://3ea5e53b-817e-4c41-ae0b-c5afc1610f4e-bluemix.cloudant.com/test/_all_docs?include_docs=true'
}
$Result = Invoke-RestMethod #Params | ConvertTo-Json -Depth 9
$Result | Out-File "D:\file.json"
$context=New-AzStorageContext -StorageAccountName "andyprivate" -StorageAccountKey ""
Set-AzStorageBlobContent -File "D:\file.json" `
-Container "" `
-Blob "file.json" `
-Context $context `
-StandardBlobTier Hot
Directly upload to Azure blob
$Params = #{
"URI" = 'https://3ea5e53b-817e-4c41-ae0b-c5afc1610f4e-bluemix.cloudant.com/test/_all_docs?include_docs=true'
}
$Result = Invoke-RestMethod #Params | ConvertTo-Json -Depth 9
Write-Host "the result is :"
$Result
$context=New-AzStorageContext -StorageAccountName "andyprivate" -StorageAccountKey ""
$container=Get-AzStorageContainer -Name "input" -Context $context
$content = [system.Text.Encoding]::UTF8.GetBytes($Result)
$container.CloudBlobContainer.GetBlockBlobReference("my.json").UploadFromByteArray($content,0,$content.Length)

Can you download an Azure Blob using Virtual Machine Identity Access Token through PowerShell without using Access Keys

I'm setting up a VM to do some bootstrapping on creation.
Part of this is to download a blob from an azure storage account to the VM.
These are all in the same subscription, resource group, etc.
I can do it this way fine:
function Get-BlobUsingVMIdentity
{
param(
[Parameter(Mandatory = $true)] $containerName,
[Parameter(Mandatory = $true)] $blobName,
[Parameter(Mandatory = $true)] $outputFolder
)
write-host "Defining package information"
mkdir $outputFolder -force
write-host "Getting Instance meta data"
$instanceInfo = Invoke-WebRequest -UseBasicParsing -Uri 'http://169.254.169.254/metadata/instance/?api-version=2018-02-01' `
-Headers #{Metadata="true"} `
| select -expand content `
| convertfrom-json `
| select -expand compute
$storageAccountName = "$($instanceInfo.resourceGroupName.replace('-rg',''))sa" # This is custom since we know our naming schema
$resourceGroupName = $($instanceInfo.resourceGroupName)
$subscriptionId = $($instanceInfo.subscriptionId)
write-host "Got storageAccountName [$storageAccountName], resourceGroupName [$resourceGroupName], subscriptionId [$subscriptionId]"
write-host "Getting VM Instance Access Token"
$response = Invoke-WebRequest -UseBasicParsing -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' `
-Headers #{Metadata = "true" }
$content = $response.Content | ConvertFrom-Json
$access_token = $content.access_token
write-host "Getting SAS Token From Storage Account"
$params = #{canonicalizedResource = "/blob/$($storageAccountName)/$($containerName)"; signedResource = "c"; signedPermission = "rcw"; signedProtocol = "https"; signedExpiry = "2031-09-23T00:00:00Z" }
$jsonParams = $params | ConvertTo-Json
$sasResponse = Invoke-WebRequest -UseBasicParsing -Uri "https://management.azure.com/subscriptions/$($subscriptionId)/resourceGroups/$($resourceGroupName)/providers/Microsoft.Storage/storageAccounts/$($storageAccountName)/listServiceSas/?api-version=2017-06-01" `
-Method POST `
-Body $jsonParams `
-Headers #{Authorization="Bearer $access_token"}
$sasContent = $sasResponse.Content | ConvertFrom-Json
$sasCred = $sasContent.serviceSasToken
write-host "Manually download blob"
$params = #{signedResource = "c"; signedPermission = "rcw"; signedProtocol = "https"; signedExpiry = "2031-09-23T00:00:00Z" }
$jsonParams = $params | ConvertTo-Json
$sasResponse = Invoke-WebRequest -UseBasicParsing -Uri "https://$($storageAccountName).blob.core.windows.net/$($containerName)/$($blobName)?api-version=2017-06-01" `
-Method POST `
-Body $jsonParams `
-Headers #{Authorization="Bearer $access_token"}
$sasContent = $sasResponse.Content | ConvertFrom-Json
$sasCred = $sasContent.serviceSasToken
write-host "Setting up storage context"
$ctx = New-AzStorageContext -StorageAccountName $storageAccountName -SasToken $sasCred
write-host "Downloading package"
Get-AzStorageBlobContent `
-Blob $blobName `
-Container $containerName `
-Destination $outputFolder `
-Context $ctx `
-Force
}
This works fine, except I have to grant full/write access to the identity in order for it to use the access key.
Is there a similar approach that would allow read only access to the blob?
My goals are:
1. No credentials stored anywhere
2. Download blob to VM from azure storage
3. No statically defined variables (ex: subscriptionid)
4. Read only access to the blob/storage account.
Appreciate any help!
According to my understanding, you want to use Azure VM MSI to access Azure storage. If so, please refer to the following steps:
Enable a system-assigned managed identity on a VM
Connect-AzAccount
$vm = Get-AzVM -ResourceGroupName myResourceGroup -Name myVM
Update-AzVM -ResourceGroupName myResourceGroup -VM $vm -AssignIdentity:$SystemAssigned
Grant your VM access to an Azure Storage container
Connect-AzAccount
$spID = (Get-AzVM -ResourceGroupName myRG -Name myVM).identity.principalid
New-AzRoleAssignment -ObjectId $spID -RoleDefinitionName "Storage Blob Data Reader" -Scope "/subscriptions/<mySubscriptionID>/resourceGroups/<myResourceGroup>/providers/Microsoft.Storage/storageAccounts/<myStorageAcct>/blobServices/default/containers/<container-name>"
access blob
# get AD access token
$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' `
-Headers #{Metadata="true"}
$content =$response.Content | ConvertFrom-Json
$access_token = $content.access_token
# call Azure blob rest api
$url="https://<myaccount>.blob.core.windows.net/<mycontainer>/<myblob>"
$RequestHeader = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$RequestHeader.Add("Authorization", "Bearer $access_token")
$RequestHeader.Add("x-ms-version", "2019-02-02")
$result = Invoke-WebRequest -Uri $url -Headers $RequestHeader
$result.content
Update
According to my test, when we get token to access Azure blob, we need to change resouce as https://storage.azure.com/
# get AD access token
$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://storage.azure.com/' `
-Headers #{Metadata="true"}
$content =$response.Content | ConvertFrom-Json
$access_token = $content.access_token
# call Azure blob rest api
$url="https://<myaccount>.blob.core.windows.net/<mycontainer>/<myblob>"
$RequestHeader = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$RequestHeader.Add("Authorization", "Bearer $access_token")
$RequestHeader.Add("x-ms-version", "2019-02-02")
$result = Invoke-WebRequest -Uri $url -Headers $RequestHeader
$result.content
Update
Found a much easier way to do it!
The key was to use:
Connect-AzAccount -identity
This auto logs in a the identity and allows you to interact through AZ module instead of trying to hack API calls together.
You only need [Storage Blob Data Reader] assigned to the container to pull this off.
Working example looks like this:
function Get-BlobUsingVMIdentity
{
param(
[Parameter(Mandatory = $true)] $containerName,
[Parameter(Mandatory = $true)] $blobName,
[Parameter(Mandatory = $true)] $outputFolder
)
write-host "Get instance info"
$instanceInfo = Invoke-WebRequest -UseBasicParsing -Uri 'http://169.254.169.254/metadata/instance/?api-version=2018-02-01' `
-Headers #{Metadata="true"} `
| select -expand content `
| convertfrom-json `
| select -expand compute
$storageAccountName = "$($instanceInfo.resourceGroupName.replace('-rg',''))sa" # This is custom since we know our naming schema
write-host "Clear existing identies to keep cache fresh"
Clear-AzContext -force
write-host "Authenticate using identity"
$account = Connect-AzAccount -identity
if(-not $account.Context.Subscription.Id)
{
write-error "Failed to authenticate with identity. Ensure VM has identity enabled and is assigned the correct IAM roles"
return
}
write-host "Get storage context"
$ctx = New-AZStorageContext -StorageAccountName $storageAccountName
write-host "Getting blob"
Get-AzStorageBlobContent `
-Blob $blobName `
-Container $containerName `
-Destination $outputFolder `
-Context $ctx `
-Force
}
Get-BlobUsingVMIdentity `
-containerName "deploy" `
-blobName "deploy.zip" `
-outputFolder "c:\deploy\"

Exception in renaming file name in ADLS Gen2 through PowerShell

I’m able to create files and folders in ADLS using PowerShell and ADLS Gen 2 REST API. However I’m having trouble renaming the file. I am using “x-ms-rename-source” in the header but its throwing exception.
code:
$n = '`n'
$stringToSign +=
#SECTION: CanonicalizedHeaders + “\n” #
“x-ms-date:$date” + $n +
“x-ms-version:2018-11-09” + $n +
“x-ms-rename-source:/adlsg2filesystemname/folderpath/filename” + $n
$stringToSign +=
# SECTION: CanonicalizedResource + “\n” #
“/$StorageAccountName/$FilesystemName” + $PathToCreate + $n
$sharedKey = [System.Convert]::FromBase64String($AccessKey)
$hasher = New-Object System.Security.Cryptography.HMACSHA256
$hasher.Key = $sharedKey
$signedSignature = [System.Convert]::ToBase64String($hasher.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stringToSign)))
$authHeader = “SharedKey ${StorageAccountName}:$signedSignature”
$headers = #{“x-ms-date”=$date}
$headers.Add(“x-ms-version”,”2018-11-09″)
$headers.Add(“x-ms-rename-source”,”/adlsg2filesystemname/folderpath/filename”)
$headers.Add(“Authorization”,$authHeader)
$headers.Add(“If-None-Match”,”*”) # To fail if the destination already exists, use a conditional request with If-None-Match: “*”
$URI = “https://$StorageAccountName.dfs.core.windows.net/” + $FilesystemName + $PathToCreate
I am getting below exception:
Invoke-RestMethod : {"error":{"code":"AuthenticationFailed","message":"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly
including the signature.\nRequestId:ddfd851b-501f-0057-3f88-7e0a7d000000\nTime:2019-10-09T09:59:53.7708781Z"}}
Any help will be truly appreciated. Thanks.
According to my test, we can use Azure AD authentication to call Azure data lake storage Gen2 REST API. For more details, please refer to https://social.msdn.microsoft.com/Forums/en-US/45be0931-379d-4252-9d20-164261cc64c5/error-while-calling-adls-gen-2-rest-api-to-create-file?forum=AzureDataLake.
Create Azure AD service principal and assign a RABC role to it. For further information, please refer to https://learn.microsoft.com/en-us/azure/storage/common/storage-auth-aad.
Connect-AzAccount
$password=''
$credentials = New-Object Microsoft.Azure.Commands.ActiveDirectory.PSADPasswordCredential -Property #{ StartDate=Get-Date; EndDate=Get-Date -Year 2024; Password=$password}
$sp = New-AzAdServicePrincipal -DisplayName jimtest1 -PasswordCredential $credentials
New-AzRoleAssignment -ApplicationId $sp.ApplicationId -RoleDefinitionName "Storage Blob Data Owner" -Scope "your scope such as your storage account scope"
get access token
$TeantID='hanxia.onmicrosoft.com'
$TokenResult = Invoke-RestMethod -Method Post -ContentType 'application/x-www-form-urlencoded' -Uri "https://login.microsoftonline.com/$($TeantID)/oauth2/token" -Body #{
client_id = $sp.ApplicationId # the application id of service principal
resource = 'https://storage.azure.com'
grant_type = 'client_credentials'
client_secret = $password # you use it in step 1
}
Call the rest api
$StorageAccountName =''
$FilesystemName =''
$PathToCreate=''
$URI = “https://$StorageAccountName.dfs.core.windows.net/” + $FilesystemName +"/"+$PathToCreate
Invoke-RestMethod -Method Put -Uri $URI -Headers #{
'Authorization' = "Bearer "+ $TokenResult.access_token
'x-ms-rename-source' = ' '
}

Resources