How to write data in Azure Blob storage programmatically? - azure

I am using below PowerShell script to read JSON data using REST API call from source. Now I want to load the data of $Result to the Azure Blob Storage. Any idea please?
$Params = #{
"URI" = 'https://3ea5e53b-817e-4c41-ae0b-c5afc1610f4e-bluemix.cloudant.com/test/_all_docs?include_docs=true'
}
$Result = Invoke-RestMethod #Params | ConvertTo-Json -Depth 9

Regarding the issue, you can use the following ways
Save the JSON into one file then upload the file to Azure blob
$Params = #{
"URI" = 'https://3ea5e53b-817e-4c41-ae0b-c5afc1610f4e-bluemix.cloudant.com/test/_all_docs?include_docs=true'
}
$Result = Invoke-RestMethod #Params | ConvertTo-Json -Depth 9
$Result | Out-File "D:\file.json"
$context=New-AzStorageContext -StorageAccountName "andyprivate" -StorageAccountKey ""
Set-AzStorageBlobContent -File "D:\file.json" `
-Container "" `
-Blob "file.json" `
-Context $context `
-StandardBlobTier Hot
Directly upload to Azure blob
$Params = #{
"URI" = 'https://3ea5e53b-817e-4c41-ae0b-c5afc1610f4e-bluemix.cloudant.com/test/_all_docs?include_docs=true'
}
$Result = Invoke-RestMethod #Params | ConvertTo-Json -Depth 9
Write-Host "the result is :"
$Result
$context=New-AzStorageContext -StorageAccountName "andyprivate" -StorageAccountKey ""
$container=Get-AzStorageContainer -Name "input" -Context $context
$content = [system.Text.Encoding]::UTF8.GetBytes($Result)
$container.CloudBlobContainer.GetBlockBlobReference("my.json").UploadFromByteArray($content,0,$content.Length)

Related

powershell upload blob to azure blob storage set content type

I am making a ReST api call and uploading the JSON payload to azure blob storage using the below powershell.
PoSh:
$Params = #{"URI" = 'https://myapiexample.com/api/data/'}
$Result = Invoke-RestMethod #Params | ConvertTo-Json
$context=New-AzStorageContext -StorageAccountName "mystorage" -StorageAccountKey ""
$container=Get-AzStorageContainer -Name "input" -Context $context
$content = [system.Text.Encoding]::UTF8.GetBytes($Result)
$blobpath = "azblob/mydata.json"
$container.CloudBlobContainer.GetBlockBlobReference($blobpath).UploadFromByteArray($content,0,$content.Length)
$container.CloudBlobContainer.GetBlockBlobReference($blobpath).Properties.ContentType = "application/json"
$container.CloudBlobContainer.GetBlockBlobReference($blobpath).SetProperties()
I notice that when the blob is stored in azure blob storage the content-type is application/octet-stream whereas I would want it to be application/json. The code I use is not working as expected.
After reproducing from my end, I could able to achieve your requirement using Set-AzStorageBlobContent. Below is the complete code that is working for me that sets the content type of the blob.
$url = "<YOUR_URL>"
$dest = "samplex.json"
Invoke-WebRequest -Uri $url -OutFile $dest
$context = New-AzStorageContext -StorageAccountName "<YOUR_ACCOUNT_NAME>" -StorageAccountKey "<YOUR_ACCOUNT_KEY>"
Set-AzStorageBlobContent -Context $context -Container "containers" -Blob "mydata.json" -File $dest -Properties #{"ContentType" = "application/json"};
RESULTS:

Undeleting a Soft Deleted Blob in Azure Storage Using a REST API call from PowerShell

I am trying to create a script to retrieve blobs for a given customer number from a storage account in Azure. All blobs reside in a single container, with 'actioned' blobs being soft deleted.
I can use PowerShell to display the relevant blobs, including their 'IsDeleted' status, but I understand that PowerShell doesn't have the necessary command to undelete blobs and so I'm trying to make a REST API call from the PowerShell script.
I do an inital login to the Azure platform and set a variable for an SAS token (which includes the necessary permissions to undelete):
$username = "<myUserName>"
$encryptedPwd = Get-Content <path\securepassword.txt> | ConvertTo-SecureString
$cred = New-Object System.Management.Automation.PsCredential($username, $encryptedPwd)
$strgaccname = "<myStorageAccount>"
$strgcontainer = "<myContainer>"
#SAS Token
$sastkn = "<mySAStoken>"
#Set StorageContext
$ctx = New-AzStorageContext -StorageAccountName $strgaccname -SasToken $sastkn
$subId = "mySubscriptionID"
Connect-AzAccount -Credential $cred -Subscription $subID
I can list all matching blobs with the following PowerShell:
$searchstring = '*'+<myCustomerNumber>+'*'
Get-AzStorageBlob -Blob $searchstring -Context $ctx -Container $strgcontainer -IncludeDeleted `
| Select-Object Name, Length, LastModified, IsDeleted `
| Sort-Object LastModified -Descending
I am unsure how to proceed with the REST API call. Looking at some other people's methods, I have something like the following, using a test blob that has been soft deleted:
$uri = "https://<myStorageAccount>.blob.core.windows.net/<myContainer>/<myTestBlob>?comp=undelete"
$headers = #{
'Authorization' = "Bearer <accessToken>";
'x-ms-date' = $((get-date -format r).ToString());
'x-ms-version' = "2020-12-06";
}
Invoke-RestMethod -Method 'Put' -Uri $uri -Headers $headers
However, I don't know how to create the Bearer Access Token that is mentioned.
We have done a repro in our local environment & it is working fine, Below statements are based on our analysis.
You can use the below Powershell script which will help you in restoring the soft-deleted blobs in your storage account.
Here is the Powershell Script :
Connect-AzAccount
#Get all deleted blob within a container
$StorageAccount = Get-AzStorageAccount | Where-Object { $_.StorageAccountName -eq "<storageAccountName>" }
$Blobs = Get-AzStorageContainer -Name "<ContainerName>" -Context $StorageAccount.Context | Get-AzStorageBlob -IncludeDeleted
$DeletedBlobs=$($Blobs| Where-Object {$_.IsDeleted -eq $true})
#Get your Bearer access token
$resource = “https://storage.azure.com"
$context = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile.DefaultContext
$accessToken = [Microsoft.Azure.Commands.Common.Authentication.AzureSession]::Instance.AuthenticationFactory.Authenticate($context.Account, $context.Environment, $context.Tenant.Id.ToString(), $null, [Microsoft.Azure.Commands.Common.Authentication.ShowDialog]::Never, $null, $resource).AccessToken
#Restore
foreach ($DeletedBlob in $DeletedBlobs) {
Write-Host "Restoring : $($DeletedBlob.Name)"
$uri = "$($DeletedBlob.BlobBaseClient.Uri.AbsoluteUri)?comp=undelete"
$headers = #{
'Authorization' = "Bearer $accessToken";
'x-ms-date' = $((get-date -format r).ToString());
'x-ms-version' = "2020-12-06";
}
Invoke-RestMethod -Method 'Put' -Uri $uri -Headers $headers
}
Here is the Sample output for your reference:
Note:
In order to perform the restoration of soft-deleted blob, you need to have a Storage Blob Data Contributor RBAC role on the Storage Account.

Azure DB sync logs in log analytics workspace

I have an Azure SQL database sync group that is scheduled to run each hour.
Question:
Can i sent this logs to a log analytics workspace by enabling the diagnostic settings?
If yes, what will be the best way to filter them out?
I can successfully get the logs from powershell, but my end goal here is to create an alert based on the sync logs.
Thanks you in advance!
If you want to send Azure SQL database sync group to a log analytics workspace, you can implement it with HTTP Data Collector API.
For example
$SubscriptionId = "SubscriptionId"
$DS_ResourceGroupName = ""
$DS_ServerName = ""
$DS_DatabaseName = ""
$DS_SyncGroupName = ""
# Replace with your OMS Workspace ID
$CustomerId = "OMSCustomerID"
# Replace with your OMS Primary Key
$SharedKey = "SharedKey"
# Specify the name of the record type that you'll be creating
$LogType = "DataSyncLog"
# Specify a field with the created time for the records
$TimeStampField = "DateValue"
Connect-AzureRmAccount
select-azurermsubscription -SubscriptionId $SubscriptionId
#get log
$endtime =[System.DateTime]::UtcNow
$StartTime = ""
$Logs = Get-AzureRmSqlSyncGroupLog -ResourceGroupName $DS_ResourceGroupName `
-ServerName $DS_ServerName `
-DatabaseName $DS_DatabaseName `
-SyncGroupName $DS_SyncGroupName `
-starttime $StartTime `
-endtime $EndTime;
if ($Logs.Length -gt 0)
{
foreach ($Log in $Logs)
{
$Log | Add-Member -Name "SubscriptionId" -Value $SubscriptionId -MemberType NoteProperty
$Log | Add-Member -Name "ResourceGroupName" -Value $DS_ResourceGroupName -MemberType NoteProperty
$Log | Add-Member -Name "ServerName" -Value $DS_ServerName -MemberType NoteProperty
$Log | Add-Member -Name "HubDatabaseName" -Value $DS_DatabaseName -MemberType NoteProperty
$Log | Add-Member -Name "SyncGroupName" -Value $DS_SyncGroupName -MemberType NoteProperty
#Filter out Successes to Reduce Data Volume to OMS
#Include the 5 commented out line below to enable the filter
#For($i=0; $i -lt $Log.Length; $i++ ) {
# if($Log[$i].LogLevel -eq "Success") {
# $Log[$i] =""
# }
# }
}
$json = ConvertTo-JSON $logs
$result = Post-OMSData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($json)) -logType $logType
if ($result -eq 200)
{
Write-Host "Success"
}
if ($result -ne 200)
{
throw
#"
Posting to OMS Failed
Runbook Name: DataSyncOMSIntegration
"#
}
Function Build-Signature ($customerId, $sharedKey, $date, $contentLength, $method, $contentType, $resource)
{
$xHeaders = "x-ms-date:" + $date
$stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource
$bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
$keyBytes = [Convert]::FromBase64String($sharedKey)
$sha256 = New-Object System.Security.Cryptography.HMACSHA256
$sha256.Key = $keyBytes
$calculatedHash = $sha256.ComputeHash($bytesToHash)
$encodedHash = [Convert]::ToBase64String($calculatedHash)
$authorization = 'SharedKey {0}:{1}' -f $customerId,$encodedHash
return $authorization
}
# Create the function to create and post the request
Function Post-OMSData($customerId, $sharedKey, $body, $logType)
{
$method = "POST"
$contentType = "application/json"
$resource = "/api/logs"
$rfc1123date = [DateTime]::UtcNow.ToString("r")
$contentLength = $body.Length
$signature = Build-Signature `
-customerId $customerId `
-sharedKey $sharedKey `
-date $rfc1123date `
-contentLength $contentLength `
-method $method `
-contentType $contentType `
-resource $resource
$uri = "https://" + $customerId + ".ods.opinsights.azure.com" + $resource + "?api-version=2016-04-01"
$headers = #{
"Authorization" = $signature;
"Log-Type" = $logType;
"x-ms-date" = $rfc1123date;
"time-generated-field" = $TimeStampField;
}
$response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing
return $response.StatusCode
}
After doing that, you can alert vai log search in Azure log analysis. For more detail, please refer to the blog.

Can you download an Azure Blob using Virtual Machine Identity Access Token through PowerShell without using Access Keys

I'm setting up a VM to do some bootstrapping on creation.
Part of this is to download a blob from an azure storage account to the VM.
These are all in the same subscription, resource group, etc.
I can do it this way fine:
function Get-BlobUsingVMIdentity
{
param(
[Parameter(Mandatory = $true)] $containerName,
[Parameter(Mandatory = $true)] $blobName,
[Parameter(Mandatory = $true)] $outputFolder
)
write-host "Defining package information"
mkdir $outputFolder -force
write-host "Getting Instance meta data"
$instanceInfo = Invoke-WebRequest -UseBasicParsing -Uri 'http://169.254.169.254/metadata/instance/?api-version=2018-02-01' `
-Headers #{Metadata="true"} `
| select -expand content `
| convertfrom-json `
| select -expand compute
$storageAccountName = "$($instanceInfo.resourceGroupName.replace('-rg',''))sa" # This is custom since we know our naming schema
$resourceGroupName = $($instanceInfo.resourceGroupName)
$subscriptionId = $($instanceInfo.subscriptionId)
write-host "Got storageAccountName [$storageAccountName], resourceGroupName [$resourceGroupName], subscriptionId [$subscriptionId]"
write-host "Getting VM Instance Access Token"
$response = Invoke-WebRequest -UseBasicParsing -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' `
-Headers #{Metadata = "true" }
$content = $response.Content | ConvertFrom-Json
$access_token = $content.access_token
write-host "Getting SAS Token From Storage Account"
$params = #{canonicalizedResource = "/blob/$($storageAccountName)/$($containerName)"; signedResource = "c"; signedPermission = "rcw"; signedProtocol = "https"; signedExpiry = "2031-09-23T00:00:00Z" }
$jsonParams = $params | ConvertTo-Json
$sasResponse = Invoke-WebRequest -UseBasicParsing -Uri "https://management.azure.com/subscriptions/$($subscriptionId)/resourceGroups/$($resourceGroupName)/providers/Microsoft.Storage/storageAccounts/$($storageAccountName)/listServiceSas/?api-version=2017-06-01" `
-Method POST `
-Body $jsonParams `
-Headers #{Authorization="Bearer $access_token"}
$sasContent = $sasResponse.Content | ConvertFrom-Json
$sasCred = $sasContent.serviceSasToken
write-host "Manually download blob"
$params = #{signedResource = "c"; signedPermission = "rcw"; signedProtocol = "https"; signedExpiry = "2031-09-23T00:00:00Z" }
$jsonParams = $params | ConvertTo-Json
$sasResponse = Invoke-WebRequest -UseBasicParsing -Uri "https://$($storageAccountName).blob.core.windows.net/$($containerName)/$($blobName)?api-version=2017-06-01" `
-Method POST `
-Body $jsonParams `
-Headers #{Authorization="Bearer $access_token"}
$sasContent = $sasResponse.Content | ConvertFrom-Json
$sasCred = $sasContent.serviceSasToken
write-host "Setting up storage context"
$ctx = New-AzStorageContext -StorageAccountName $storageAccountName -SasToken $sasCred
write-host "Downloading package"
Get-AzStorageBlobContent `
-Blob $blobName `
-Container $containerName `
-Destination $outputFolder `
-Context $ctx `
-Force
}
This works fine, except I have to grant full/write access to the identity in order for it to use the access key.
Is there a similar approach that would allow read only access to the blob?
My goals are:
1. No credentials stored anywhere
2. Download blob to VM from azure storage
3. No statically defined variables (ex: subscriptionid)
4. Read only access to the blob/storage account.
Appreciate any help!
According to my understanding, you want to use Azure VM MSI to access Azure storage. If so, please refer to the following steps:
Enable a system-assigned managed identity on a VM
Connect-AzAccount
$vm = Get-AzVM -ResourceGroupName myResourceGroup -Name myVM
Update-AzVM -ResourceGroupName myResourceGroup -VM $vm -AssignIdentity:$SystemAssigned
Grant your VM access to an Azure Storage container
Connect-AzAccount
$spID = (Get-AzVM -ResourceGroupName myRG -Name myVM).identity.principalid
New-AzRoleAssignment -ObjectId $spID -RoleDefinitionName "Storage Blob Data Reader" -Scope "/subscriptions/<mySubscriptionID>/resourceGroups/<myResourceGroup>/providers/Microsoft.Storage/storageAccounts/<myStorageAcct>/blobServices/default/containers/<container-name>"
access blob
# get AD access token
$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' `
-Headers #{Metadata="true"}
$content =$response.Content | ConvertFrom-Json
$access_token = $content.access_token
# call Azure blob rest api
$url="https://<myaccount>.blob.core.windows.net/<mycontainer>/<myblob>"
$RequestHeader = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$RequestHeader.Add("Authorization", "Bearer $access_token")
$RequestHeader.Add("x-ms-version", "2019-02-02")
$result = Invoke-WebRequest -Uri $url -Headers $RequestHeader
$result.content
Update
According to my test, when we get token to access Azure blob, we need to change resouce as https://storage.azure.com/
# get AD access token
$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://storage.azure.com/' `
-Headers #{Metadata="true"}
$content =$response.Content | ConvertFrom-Json
$access_token = $content.access_token
# call Azure blob rest api
$url="https://<myaccount>.blob.core.windows.net/<mycontainer>/<myblob>"
$RequestHeader = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$RequestHeader.Add("Authorization", "Bearer $access_token")
$RequestHeader.Add("x-ms-version", "2019-02-02")
$result = Invoke-WebRequest -Uri $url -Headers $RequestHeader
$result.content
Update
Found a much easier way to do it!
The key was to use:
Connect-AzAccount -identity
This auto logs in a the identity and allows you to interact through AZ module instead of trying to hack API calls together.
You only need [Storage Blob Data Reader] assigned to the container to pull this off.
Working example looks like this:
function Get-BlobUsingVMIdentity
{
param(
[Parameter(Mandatory = $true)] $containerName,
[Parameter(Mandatory = $true)] $blobName,
[Parameter(Mandatory = $true)] $outputFolder
)
write-host "Get instance info"
$instanceInfo = Invoke-WebRequest -UseBasicParsing -Uri 'http://169.254.169.254/metadata/instance/?api-version=2018-02-01' `
-Headers #{Metadata="true"} `
| select -expand content `
| convertfrom-json `
| select -expand compute
$storageAccountName = "$($instanceInfo.resourceGroupName.replace('-rg',''))sa" # This is custom since we know our naming schema
write-host "Clear existing identies to keep cache fresh"
Clear-AzContext -force
write-host "Authenticate using identity"
$account = Connect-AzAccount -identity
if(-not $account.Context.Subscription.Id)
{
write-error "Failed to authenticate with identity. Ensure VM has identity enabled and is assigned the correct IAM roles"
return
}
write-host "Get storage context"
$ctx = New-AZStorageContext -StorageAccountName $storageAccountName
write-host "Getting blob"
Get-AzStorageBlobContent `
-Blob $blobName `
-Container $containerName `
-Destination $outputFolder `
-Context $ctx `
-Force
}
Get-BlobUsingVMIdentity `
-containerName "deploy" `
-blobName "deploy.zip" `
-outputFolder "c:\deploy\"

I want to use the invoke-restmethod in a foreach loop to upload a multiplefiles through Infile parameter

Each time the loop executes the file is being replaced in the place of the first file . I want to upload it as a new file without distrubing the existing ones..
foreach ($blob in $blobs)
{
$file=New-TemporaryFile
$file=Get-AzureStorageBlobContent -Container $container_name -Blob $blob.Name -Context $ctx -Destination $localFile -Force
$contents = Get-Content $localFile -Raw -ErrorAction:SilentlyContinue
$f=New-TemporaryFile
Add-Content $f $contents
$Header = #{
"Content-Disposition"="attachment;filename=$($blob.Name)"
"Authorization"=$accessToken
}
Invoke-RestMethod -Uri $apiUrl -Headers $Header -Method put -InFile $f
}
I think you over-did it with the New-TemporaryFile commands in your code.
Why use Get-AzureStorageBlobContent to store blob content in a file and then create another temp file to copy the contents to?
This would be much simpler:
foreach ($blob in $blobs) {
$localFile = New-TemporaryFile
Get-AzureStorageBlobContent -Container $container_name -Blob $blob.Name -Context $ctx -Destination $localFile.FullName -Force
$Header = #{
"Content-Disposition"="attachment;filename=$($blob.Name)"
"Authorization"=$accessToken
}
Invoke-RestMethod -Uri $apiUrl -Headers $Header -Method put -InFile $localFile.FullName
}
Edit
The New-TemporaryFile command returns a System.IO.FileInfo object which you cannot use directly as the InFile parameter. Instead, it expects a full path and file name, so we use $localFile.FullName

Resources