$ChildItem = Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem
$Container.Name -MaxCount $MaxReturn -ContinuationToken $Token
Foreach ($Item in $ChildItem){
$Token = $Item[$item.Count -1].ContinuationToken
Write-Host "File: $($Container.Name)"/"$($Item.Path)”
}
I tried removing the -Recurse parameter but I only get the first folder back
After reproducing from my end I could able to achieve your requirement using Where-Object. Below is the complete script that worked for me.
$ListOfFiles = Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem $filesystemName -Recurse -FetchProperty
$ListOfFiles | Where-Object { $_.Path -match "dir1/" -or $_.Path -match "dir2/"}
Results:
Related
I'm trying to convert JSON file which is present in storage account data into Powershell object. But I'm not getting the proper output. Output does not contain proper values.
My code:
$storageAccountKey = "xxxx"
$Context = New-AzStorageContext -StorageAccountName 'xxxx' -StorageAccountKey $storageAccountKey
$b='C:\Temp\Scheduled.json'
Get-AzureStorageFileContent -Path $b -ShareName "file-share-name"
$newScheduledRules = Get-Content -Raw $b | ConvertFrom-Json
Write-Output ("########newScheduledRules::###########" + $newScheduledRules)
Output:
Could not get the storage context. Please pass in a storage context or set the current storage context.
########newScheduledRules::############{Scheduled=System.Object[]; Fusion=System.Object[]; MLBehaviorAnalytics=System.Object[]; MicrosoftSecurityIncidentCreation=System.Object[]}
I have reproduced in my environment and got expected results as below and followed below process and followed Microsoft-Document:
Scheduled.json:
{
"Rithwik":"Hello",
"Chotu":"Bojja"
}
Firstly, I have created Storage account and then added a Scheduled.json in file share as below:
Now i have created a runbook and excuted below script in runbook as below:
$storageAccountKey = "PFHxFbVmAEvwBM6/9kW4nORJYA+AStA2QQ1A=="
$Context = New-AzStorageContext -StorageAccountName 'rithwik' -StorageAccountKey $storageAccountKey
$out = "$($env:TEMP)\$([guid]::NewGuid())"
New-Item -Path $out -ItemType Directory -Force
Get-AzStorageFileContent -ShareName "rithwik" -Context $Context -Path 'Scheduled.json' -Destination $out -Force
$newScheduledRules = Get-Content -Path "$out\Scheduled.json" -Raw | ConvertFrom-Json
Write-Output ("########newScheduledRules::###########" + $newScheduledRules)
Output:
Here $out is the Destination Variable.
-Path should be Only the file name Scheduled.json in
Get-AzStorageFileContent command.
It seems the Get-AzureStorageFileContent is missing -Context parameter. It should be something like this
$OutPath = "$($env:TEMP)\$([guid]::NewGuid())"
New-Item -Path $OutPath -ItemType Directory -Force
$storageContext = (Get-AzStorageAccount -ResourceGroupName xxxx -Name xxxx).Context
Get-AzStorageFileContent -ShareName "file-share-name" -Context $storageContext -Path 'Scheduled.json' -Destination $OutPath -Force
$newScheduledRules = Get-Content -Path "$OutPath\Scheduled.json" -Raw | ConvertFrom-Json
Write-Output ("########newScheduledRules::###########" + $newScheduledRules)
I am trying to delete file from specific folder like from full or diff in blob container but unable to do.
Container_name and then there are two folders full and diff and I want to delete file from full only.
Please help.
$context = New-AzureStorageContext -StorageAccountName "storage_name" -StorageAccountKey "key"
$blobs= Get-AzureStorageBlob -Container "container_name" -blob *DIFF*.bak -Context $context
foreach ($blob in $blobs)
{
$modifieddate = $blob.LastModified
Write-Host $modifieddate
if ($modifieddate -ne $null)
{
$howold = ([DateTime]::Now - [DateTime]$modifieddate.LocalDateTime)
if ($howold.TotalDays -ge 5)
{
Remove-AzureStorageBlob -Blob $blob.Name -Container "container_name" -Context $context
Write-Host $blob.Name
}
}
}
If you want to use Azure PowerShell to delete blobs from one subfolder in one container, you can use the following script :
$StorageAccountKey=" "
$StorageAccountName=" "
$ContainerName=" "
$Token = $null
$Total = 0
$MaxCount=5000
$context = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
do
{
$Blobs = Get-AzStorageBlob -Container $ContainerName -MaxCount $MaxCount -ContinuationToken $Token -Context $context -Prefix "your fodler name"
if($Blobs.Length -le 0) { Break;}
$Token = $Blobs[$blobs.Count -1].ContinuationToken;
foreach($blob in $blobs){
Remove-AzStorageBlob -Blob $blob.Name -Container $ContainerName -Context $context
}
}
While ($Token -ne $null)
Update
I use the latest version Azure Az module to do a test.
How do I retrieve the metadata of a blob folder's contents (such as name, size etc) with powershell and export it as CSV?
Assume you have azure powershell az module installed(also you can change the code accordingly if you are using AzureRm module).
There are 2 methods:
Method 1: Use PSCustomObject. Sample code as below:
$accountname="xxx"
$accountkey="xxx"
$ctx = New-AzStorageContext -StorageAccountName $accountname -StorageAccountKey $accountkey
$myblobs = Get-AzStorageBlob -Container "aa1" -Context $ctx
$Output = $myblobs | ForEach-Object {
[PSCustomObject]#{
"Name" = $_.Name
"Length" =$_.Length
}
}
$Output | Export-Csv "d:\temp\test2.csv" -NoTypeInformation
Method 2: Use psobject. Sample code as below, save the generated file to location "d:\temp\test2.csv":
$accountname="xxx"
$accountkey="xxx"
$ctx = New-AzStorageContext -StorageAccountName $accountname -StorageAccountKey $accountkey
$myblobs = Get-AzStorageBlob -Container "aa1" -Context $ctx
$i=0
foreach($b in $myblobs)
{
$i++
if($i -eq 1)
{
$obj = New-Object psobject
$obj | add-member -membertype NoteProperty -name "Name" -value $b.name
#the length unit is byte
$obj | add-member -membertype NoteProperty -name "Length" -value $b.length
$obj | Export-Csv "d:\temp\test2.csv" -NoTypeInformation
}
else
{
$obj = New-Object psobject
$obj | add-member -membertype NoteProperty -name "Name" -value $b.name
$obj | add-member -membertype NoteProperty -name "Length" -value $b.length
$obj | Export-Csv "d:\temp\test2.csv" -NoTypeInformation -Append
}
}
Test result as below:
I have got a problem to set my content in AzureBlobStorage.
In local, I have succeeded to replace characters for each files in a directory.
$sourceFolder = "C:\MyDirectory"
$targetFolder = "C:\MyDirectoryEncodeded"
$fileList = Dir $sourceFolder -Filter *.dat
MkDir $targetFolder -ErrorAction Ignore
ForEach($file in $fileList) {
$file | Get-Content | %{$_ -replace '"',''} | %{$_ -replace ',','.'} | Set-Content -Path "tempDirectory\$file"
$newFile = Get-Content "tempDirectory\$file"
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
[System.IO.File]::WriteAllLines("targetDirectory\$file" , $newFile,$Utf8NoBomEncoding)
}
exit
But now, I need to do the same in Microsoft Azure.
I get the content into an Azure Blob Storage, I escape characters, I encoding my file in UTF-8NoBom and then I set the encode file into a new Blob Directory.
Nevertheless, I faced an issue when I want to set the new content with escape characters (First line in my loop).
$storageContext = New-AzureStorageContext -ConnectionString "DefaultEndpointsProtocol=https;AccountName=<myAccountName>;AccountKey=<myAccountKey>;"
$sourceFolder = Get-AzureStorageBlob -Container "datablobnotencoded" -Blob "*.dat" -Context $storageContext
$targetFolder = Get-AzureStorageBlob -Container "datablob" -Context $storageContext
MkDir $targetFolder -ErrorAction Ignore
ForEach($file in $sourceFolder) {
Get-AzureStorageBlob -Container "datablobnotencoded" -Blob $file.Name -Context $storageContext | Get-AzureStorageBlobContent | %{$_ -replace '"',''} | %{$_ -replace ',','.'} | Set-AzureStorageBlobContent -File $file.Name -Context $storageContext -CloudBlob $file
$newFile = Get-AzureStorageFileContent -Path $file
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
[System.IO.File]::WriteAllLines($file , $newFile, $Utf8NoBomEncoding)
}
I've got this error:
Set-AzureStorageBlobContent : Cannot bind parameter 'CloudBlob'.
Cannot convert the
"Microsoft.WindowsAzure.Commands.Storage.Model.ResourceModel.AzureStorageBlob"
value of type
"Microsoft.WindowsAzure.Commands.Storage.Model.ResourceModel.AzureStorageBlob"
to type "Microsoft.WindowsAzure.Storage.Blob.CloudBlob". At line:7
char:264
+ ... lobContent -File $file.Name -Context $storageContext -CloudBlob $file
+ ~~~~~
+ CategoryInfo : InvalidArgument: (:) [Set-AzureStorageBlobContent], ParameterBindingException
+ FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand
Thank you for your answers!
There are some mistakes in your powershell scripts:
1.You may misunderstand the usage of Get-AzureStorageBlobContent, it's used to download blob to local, you cann't get the content of the blob, more details refer here.
2.In the loop, you used $newFile = Get-AzureStorageFileContent -Path $file, the Get-AzureStorageFileContent cmdlet is for file share storage, not for the blob storage.
You can use Get-AzureStorageBlobContent to download the blobs to a local folder, then operate on the local file which is downloaded from blob storage. After the file is modified, you can use Set-AzureStorageBlobContent to upload the local files to the specified azure blob storage.
Sample code as below, and works fine at my side:
$context = New-AzureStorageContext -ConnectionString "xxxx"
#download the blobs in specified contianers
$sourceFolder_blob = Get-AzureStorageBlob -Container "test-1" -Blob "*.txt" -Context $context
#the target azure container, which you want to upload the modifed blob to
$taget_container="test-2"
#the local path which is used to store the download blobs, and make sure the folders exist before use.
$sourceFolder_local="d:\test\blob1\"
$targetFolder_local="d:\test\blob2\"
foreach($file in $sourceFolder_blob)
{
#download the specified blob to local path
Get-AzureStorageBlobContent -Container "test-1" -Blob $file.name -Destination $sourceFolder_local -Context $context
#get the local file path
$local_file_path=$sourceFolder_local + $file.name
#set content to the file in target local folder
$local_target_file_path = "$targetFolder_local"+$file.name
#since the files are downloaded to local, you can any operation for the local file
Get-Content $local_file_path | %{$_ -replace '-','!'} | %{$_ -replace ',','.'} | Set-Content -Path $local_target_file_path
$newFile = Get-Content -Path $local_target_file_path
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
[System.IO.File]::WriteAllLines($local_target_file_path , $newFile,$Utf8NoBomEncoding)
#the last step, upload the modified file to another azure container
Set-AzureStorageBlobContent -File $local_target_file_path -Context $context -Container $taget_container
}
We have multiple blobs in an azure storage container, when we use powershell to download the blobs (files), 8 out of the 9 files download, however the 9th one fails. There is absolutely nothing different about this file, the only thing I've noticed in the blob properties the "content MD5" is blank, however the other 8 have a value. Not sure what this is or if it has anything to do wit it, I was hoping someone could shed some light as to why this is one file is not downloading..
Thanks in advance :)
Try below code for downloading files from Azure Blob
function Get-DLLFile
{
param(
[Parameter(Mandatory=$true)] [string] $connectionString,
[Parameter(Mandatory=$true)] [String[]] $blobsName,
[Parameter(Mandatory=$true)] [string] $container,
[Parameter(Mandatory=$true)] [string] $filePath
)
Try
{
foreach ($blobName in $blobsName)
{
$file = $filePath + $blobName
$fileAvailable = Get-Item -Path $file -ErrorAction SilentlyContinue
if($null -eq $fileAvailable)
{
$ctx = New-AzureStorageContext -ConnectionString $connectionString
New-Item -Path $filePath -ItemType Directory -Force | Out-Null
Get-AzureStorageBlobContent -Blob $blobName -Container $container -Destination $filePath -Context $ctx -Force | out-null
}
}
}
Catch
{
$_.Exception.Message
}
}
Get-DLLFile -blobsName "File1.csv","File2.json" -container "myContainer" -connectionString "$(BlobConnectionString)" -filePath "$(System.DefaultWorkingDirectory)/Download"
Hope this should work, if not Please share the exception you are getting.