I'm trying to copy-Blob from azure storage for that i have taken a runbook from the Azure runbook gallery named "Copy-BlobFromAzureStorage". When i try to test, it prompt me for "PATHTOPLACEBLOB" here i have given the default location "c:/" . and its running fine.But the thing is I don't understand where exactly i can find the stored blob, and it is given " PSComputerName" as "localhost". Can any one please suggest me regarding this.
Code:
workflow Copy-BlobFromAzureStorage{
param
(
[parameter(Mandatory=$True)]
[String]
$AzureSubscriptionName,
[parameter(Mandatory=$True)]
[PSCredential]
$AzureOrgIdCredential,
[parameter(Mandatory=$True)]
[String]
$StorageAccountName,
[parameter(Mandatory=$True)]
[String]
$ContainerName,
[parameter(Mandatory=$True)]
[String]
$BlobName,
[parameter(Mandatory=$False)]
[String]
$PathToPlaceBlob = "C:\"
)
$Null = Add-AzureAccount -Credential $AzureOrgIdCredential
$Null = Select-AzureSubscription -SubscriptionName $AzureSubscriptionName
Write-Verbose "Downloading $BlobName from Azure Blob Storage to $PathToPlaceBlob"
Set-AzureSubscription `
-SubscriptionName $AzureSubscriptionName `
-CurrentStorageAccount $StorageAccountName
$blob =
Get-AzureStorageBlobContent `
-Blob $BlobName `
-Container $ContainerName `
-Destination $PathToPlaceBlob `
-Force
try {
Get-Item -Path "$PathToPlaceBlob\$BlobName" -ErrorAction Stop
}
catch {
Get-Item -Path $PathToPlaceBlob
}}
The blob is placed on the sandbox where the Azure Automation runbook is running. There's not much point in putting it there, since this sandbox will be cleaned up after the runbook job finishes, but it can make sense, depending on your scenario, as an intermediary point to put the blob, such as to edit it or transfer it to somewhere else outside of the sandbox (ex another Azure Storage account or an FTP server).
Related
I'm developing a new powershell script in order to download any blobs from a specific container and the problem is due to security reasons because I do not want to paste in text plain the azure account key.
So I have implemented a solution using 'ConvertTo-SecureString' command but the problem still exists because when I create a connection string to the blob, there appears a message who said: "Server Failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. HTTP Status Code 403 - HTTP Error".
With the key in plain text I'm able to create the connection string properly and then list and download all blobs from the container.
I tried other solutions for example ' $Credential= New-Object System.Management.Automation.PSCredential ('$ShareUser, $SharePassword)'
but there is other problem related with the input is not valid base64 string.
Do you know how to avoid this issues and create a secure connection string with an Azure Storage Account?
Best regards and thanks in advance
Here a part of my powershell script
$SecurePassword= Read-Host -AsSecureString | ConvertFrom-SecureString
$SecurePassword | Out-File -FilePath C:\test_blob\pass_file.xml
$ConfigFile= 'C:\Users\\config_file.xml'
IF (Test-Path) {
[xml]$Config= Get-Content $ConfigFile
[string] $Server = $Config.Config.Server;
[string] $SharePassword = $Config.Config.SharePassword;
} ELSE
{
write-host "File do not exists: $ConfigFile"
}
#BlobStorageInformation
$StorageAccountName='test_acc'
$Container='test'
$DestinationFolder= 'C:\Users\user1\Blobs'
$Context = New-AzStorageConext -StorageAccountName $StorageAccountName -StorageAccountKey $SharePassword
#List of Blobs
$ListBlob=#()
$ListBlob+= Get-AzStorageBlob -context $Context -container $Container | Where-Object {$_.LastModified -lt (Get-Date).AddDAys(-1)}
Why would you maintain the password files or enter storage key manually when you have az powershell. Just login using az powershell, set the subscription and enjoy !
$ResourceGroupName = "YOURRESOURCEGROUPNAME"
$StorageAccountName = "YOURSTORAGEACCOUNTNAME"
$ContainerName = "YOURCONTAINERNAME"
$LocalPath = "D:\Temp"
Write-Output 'Downloading Content from Azure blob to local...'
$storageKey = (Get-AzStorageAccountKey -ResourceGroupName $ResourceGroupName -AccountName $StorageAccountName).value[0]
$storageContext = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $storageKey
$blobs = Get-AzStorageBlob -Container $ContainerName -Context $storageContext
foreach($blob in $blobs)
{
Get-AzStorageBlobContent -Container $ContainerName -Context $storageContext -Force -Destination $LocalPath -Blob $blob.Name
}
Write-Output 'Content Downloaded Successfully !!!'
I want daily backup data in Azure SQL Database then save as a file in Blob and when my system has an error I can import a backup in Blob to recover the database. In the export case, I found DataFactory to do that but, It is hard to import data. What is the best way to resolve my problem?
Thanks for your help.
The best way to do this is to schedule a daily job using Azure Automation. Below you will find a PowerShell runbook you can use on Azure Automation to backup your database to an Azure storage account.
<#
.SYNOPSIS
This Azure Automation runbook automates Azure SQL database backup to Blob storage and deletes old backups from blob storage.
.DESCRIPTION
You should use this Runbook if you want manage Azure SQL database backups in Blob storage.
This runbook can be used together with Azure SQL Point-In-Time-Restore.
This is a PowerShell runbook, as opposed to a PowerShell Workflow runbook.
.PARAMETER ResourceGroupName
Specifies the name of the resource group where the Azure SQL Database server is located
.PARAMETER DatabaseServerName
Specifies the name of the Azure SQL Database Server which script will backup
.PARAMETER DatabaseAdminUsername
Specifies the administrator username of the Azure SQL Database Server
.PARAMETER DatabaseAdminPassword
Specifies the administrator password of the Azure SQL Database Server
.PARAMETER DatabaseNames
Comma separated list of databases script will backup
.PARAMETER StorageAccountName
Specifies the name of the storage account where backup file will be uploaded
.PARAMETER BlobStorageEndpoint
Specifies the base URL of the storage account
.PARAMETER StorageKey
Specifies the storage key of the storage account
.PARAMETER BlobContainerName
Specifies the container name of the storage account where backup file will be uploaded. Container will be created if it does not exist.
.PARAMETER RetentionDays
Specifies the number of days how long backups are kept in blob storage. Script will remove all older files from container.
For this reason dedicated container should be only used for this script.
.INPUTS
None.
.OUTPUTS
Human-readable informational and error messages produced during the job. Not intended to be consumed by another runbook.
#>
param(
[parameter(Mandatory=$true)]
[String] $ResourceGroupName,
[parameter(Mandatory=$true)]
[String] $DatabaseServerName,
[parameter(Mandatory=$true)]
[String]$DatabaseAdminUsername,
[parameter(Mandatory=$true)]
[String]$DatabaseAdminPassword,
[parameter(Mandatory=$true)]
[String]$DatabaseNames,
[parameter(Mandatory=$true)]
[String]$StorageAccountName,
[parameter(Mandatory=$true)]
[String]$BlobStorageEndpoint,
[parameter(Mandatory=$true)]
[String]$StorageKey,
[parameter(Mandatory=$true)]
[string]$BlobContainerName,
[parameter(Mandatory=$true)]
[Int32]$RetentionDays
)
$ErrorActionPreference = 'stop'
function Login() {
$connectionName = "AzureRunAsConnection"
try
{
$servicePrincipalConnection = Get-AutomationConnection -Name $connectionName
Write-Verbose "Logging in to Azure..." -Verbose
Add-AzureRmAccount `
-ServicePrincipal `
-TenantId $servicePrincipalConnection.TenantId `
-ApplicationId $servicePrincipalConnection.ApplicationId `
-CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint | Out-Null
}
catch {
if (!$servicePrincipalConnection)
{
$ErrorMessage = "Connection $connectionName not found."
throw $ErrorMessage
} else{
Write-Error -Message $_.Exception
throw $_.Exception
}
}
}
function Create-Blob-Container([string]$blobContainerName, $storageContext) {
Write-Verbose "Checking if blob container '$blobContainerName' already exists" -Verbose
if (Get-AzureStorageContainer -ErrorAction "Stop" -Context $storageContext | Where-Object { $_.Name -eq $blobContainerName }) {
Write-Verbose "Container '$blobContainerName' already exists" -Verbose
} else {
New-AzureStorageContainer -ErrorAction "Stop" -Name $blobContainerName -Permission Off -Context $storageContext
Write-Verbose "Container '$blobContainerName' created" -Verbose
}
}
function Export-To-Blob-Storage([string]$resourceGroupName, [string]$databaseServerName, [string]$databaseAdminUsername, [string]$databaseAdminPassword, [string[]]$databaseNames, [string]$storageKey, [string]$blobStorageEndpoint, [string]$blobContainerName) {
Write-Verbose "Starting database export to databases '$databaseNames'" -Verbose
$securePassword = ConvertTo-SecureString –String $databaseAdminPassword –AsPlainText -Force
$creds = New-Object –TypeName System.Management.Automation.PSCredential –ArgumentList $databaseAdminUsername, $securePassword
foreach ($databaseName in $databaseNames.Split(",").Trim()) {
Write-Output "Creating request to backup database '$databaseName'"
$bacpacFilename = $databaseName + (Get-Date).ToString("yyyyMMddHHmm") + ".bacpac"
$bacpacUri = $blobStorageEndpoint + $blobContainerName + "/" + $bacpacFilename
$exportRequest = New-AzureRmSqlDatabaseExport -ResourceGroupName $resourceGroupName –ServerName $databaseServerName `
–DatabaseName $databaseName –StorageKeytype "StorageAccessKey" –storageKey $storageKey -StorageUri $BacpacUri `
–AdministratorLogin $creds.UserName –AdministratorLoginPassword $creds.Password -ErrorAction "Stop"
# Print status of the export
Get-AzureRmSqlDatabaseImportExportStatus -OperationStatusLink $exportRequest.OperationStatusLink -ErrorAction "Stop"
}
}
function Delete-Old-Backups([int]$retentionDays, [string]$blobContainerName, $storageContext) {
Write-Output "Removing backups older than '$retentionDays' days from blob: '$blobContainerName'"
$isOldDate = [DateTime]::UtcNow.AddDays(-$retentionDays)
$blobs = Get-AzureStorageBlob -Container $blobContainerName -Context $storageContext
foreach ($blob in ($blobs | Where-Object { $_.LastModified.UtcDateTime -lt $isOldDate -and $_.BlobType -eq "BlockBlob" })) {
Write-Verbose ("Removing blob: " + $blob.Name) -Verbose
Remove-AzureStorageBlob -Blob $blob.Name -Container $blobContainerName -Context $storageContext
}
}
Write-Verbose "Starting database backup" -Verbose
$StorageContext = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageKey
Login
Create-Blob-Container `
-blobContainerName $blobContainerName `
-storageContext $storageContext
Export-To-Blob-Storage `
-resourceGroupName $ResourceGroupName `
-databaseServerName $DatabaseServerName `
-databaseAdminUsername $DatabaseAdminUsername `
-databaseAdminPassword $DatabaseAdminPassword `
-databaseNames $DatabaseNames `
-storageKey $StorageKey `
-blobStorageEndpoint $BlobStorageEndpoint `
-blobContainerName $BlobContainerName
Delete-Old-Backups `
-retentionDays $RetentionDays `
-storageContext $StorageContext `
-blobContainerName $BlobContainerName
Write-Verbose "Database backup script finished" -Verbose
We have multiple blobs in an azure storage container, when we use powershell to download the blobs (files), 8 out of the 9 files download, however the 9th one fails. There is absolutely nothing different about this file, the only thing I've noticed in the blob properties the "content MD5" is blank, however the other 8 have a value. Not sure what this is or if it has anything to do wit it, I was hoping someone could shed some light as to why this is one file is not downloading..
Thanks in advance :)
Try below code for downloading files from Azure Blob
function Get-DLLFile
{
param(
[Parameter(Mandatory=$true)] [string] $connectionString,
[Parameter(Mandatory=$true)] [String[]] $blobsName,
[Parameter(Mandatory=$true)] [string] $container,
[Parameter(Mandatory=$true)] [string] $filePath
)
Try
{
foreach ($blobName in $blobsName)
{
$file = $filePath + $blobName
$fileAvailable = Get-Item -Path $file -ErrorAction SilentlyContinue
if($null -eq $fileAvailable)
{
$ctx = New-AzureStorageContext -ConnectionString $connectionString
New-Item -Path $filePath -ItemType Directory -Force | Out-Null
Get-AzureStorageBlobContent -Blob $blobName -Container $container -Destination $filePath -Context $ctx -Force | out-null
}
}
}
Catch
{
$_.Exception.Message
}
}
Get-DLLFile -blobsName "File1.csv","File2.json" -container "myContainer" -connectionString "$(BlobConnectionString)" -filePath "$(System.DefaultWorkingDirectory)/Download"
Hope this should work, if not Please share the exception you are getting.
I am trying hands on on azure automation runbook, below is the small script i am using to create a storage account.
Param
(
[Parameter(Mandatory=$true)]
[String]
$AzureResourceGroup,
[Parameter(Mandatory=$true)]
[String]
$StorageAC,
[Parameter(Mandatory=$true)]
[String]
$Loc,
[Parameter(Mandatory=$true)]
[String]
$sku
)
$CredentialAssetName = "AutomationAccount";
$Cred = Get-AutomationPSCredential -Name $CredentialAssetName
if(!$Cred) {
Throw "Could not find an Automation Credential Asset named
'${CredentialAssetName}'. Make sure you have created one in this Automation
Account."
}
Add-AzureRmAccount -Credential $Cred
Add-AzureAccount -Credential $Cred
$storeac = Get-AzureRmStorageAccount -ResourceGroupName $AzureResourceGroup
if ($storeac.StorageAccountName -eq "testdd" ){
write-output " AC found !";
} else {
New-AzureRmStorageAccount -ResourceGroupName $AzureResourceGroup -Name
$StorageAC -Location $Loc -SkuName $sku
}
However, when ever i run it after publishing, the job is completed with an error
(New-AzureRmStorageAccount : A parameter cannot be found that matches parameter name 'SkuName')
Can someone tell me what am i doing wrong ??
To resolve the error :
New-AzureRmStorageAccount : A parameter cannot be found that matches parameter name 'SkuName'
Please update the Azure modules in you automation account and by clicking : "Update Azure Modules" under Automation Accounts => Modules and retry the runbook
Update Azure Modules
I'm trying to write a script that I can automate on Azure to create a new instance of a cloud service. I am having trouble getting the New-AzureDeployment cmdlet to work.
Both the CSPKG and CSCFG files are stored on Azure under the same storage account but in different containers. They were uploaded using CloudBerry.
param(
[parameter(Mandatory=$False)]
[string] $StorageAccount = 'storageaccount',
[parameter(Mandatory=$False)]
[string] $ServiceName = 'cloudservicesdev',
[parameter(Mandatory=$False)]
[string] $Slot = 'Production',
[parameter(Mandatory=$False)]
[string] $Label = 'BASE'
)
Write-Output "Start of workflow"
$cert = Get-AutomationCertificate -Name 'Credential'
$subID = 'subId'
Set-AzureSubscription -SubscriptionName "SubName" -CurrentStorageAccountName $StorageAccount -Certificate $cert -SubscriptionId $subID
Select-AzureSubscription "SubName"
$package = (Get-AzureStorageBlob -blob "package.cspkg" -Container "package").ICloudBlob.uri.AbsoluteUri
$config = (Get-AzureStorageBlob -blob "Config - Dev.cscfg" -Container "config").ICloudBlob.uri.AbsoluteUri
New-AzureDeployment -ServiceName "$ServiceName" -Slot "$Slot" -Package "$package" -Configuration "$config" -Label "$Label"
I get the following error
2014-12-08 05:57:57 PM, Error: New-AzureDeployment : The given path's format is not supported.
At Create_New_Cloud_Service_Instance:40 char:40
+
+ CategoryInfo : NotSpecified: (:) [New-AzureDeployment], NotSupportedException
+ FullyQualifiedErrorId :
System.NotSupportedException,Microsoft.WindowsAzure.Commands.ServiceManagement.HostedServices.NewAzureDeploymentCommand
I've checked both the $package and $config variables and they are pointing to the file locations I'd expect (https://storageaccount.blob.core.windows.net/package/package.cspkg and https://storageaccount.blob.core.windows.net/config/Config%20-%20Dev.cscfg respectively). They match the URLs I see when I navigate to the files under their containers in Storage.
This looks similar to the examples that I have seen. What have I done wrong?
This is the new code that I used based on Joe's answer below
I also used this example script https://gallery.technet.microsoft.com/scriptcenter/Continuous-Deployment-of-A-eeebf3a6 to help with the using the Azure Automation sandbox
param(
[parameter(Mandatory=$False)]
[string] $StorageAccount = 'storageaccount',
[parameter(Mandatory=$False)]
[string] $ServiceName = 'cloudservicesdev',
[parameter(Mandatory=$False)]
[string] $Slot = 'Production',
[parameter(Mandatory=$False)]
[string] $Label = 'BASE'
)
Write-Output "Start of workflow"
$cert = Get-AutomationCertificate -Name 'Credential'
$subID = 'subId'
Set-AzureSubscription -SubscriptionName "SubName" -CurrentStorageAccountName $StorageAccount -Certificate $cert -SubscriptionId $subID
Select-AzureSubscription "SubName"
$package = (Get-AzureStorageBlob -blob "package.cspkg" -Container "package").ICloudBlob.uri.AbsoluteUri
$TempFileLocation = "C:\temp\Config - Dev.cscfg"
$config = (Get-AzureStorageBlobContent -blob "Config - Dev.cscfg" -Container "config" -Destination $TempFileLocation -Force)
New-AzureDeployment -ServiceName "$ServiceName" -Slot "$Slot" -Package "$package" -Configuration $TempFileLocation -Label "$Label"
The -Package parameter of New-AzureDeployment should be passed a storage URI as you are doing, but the -Configuration parameter expects a local file. See http://msdn.microsoft.com/en-us/library/azure/dn495143.aspx for more details.
So you need to, within the runbook, download the file at the $config URI to the Azure Automation sandbox, and then pass that file's local path to New-AzureDeployment.