Jenkins using PowerShell Azure Modules failing to run - azure

I have a bit of a strange one here.
I have written a PowerShell script to pull Azure blob storage objects which when I run manually via the console works absolutely fine. However if i run it from jenkins, it calls the PowerShell function, starts running through it but fails when using the az.storage function Get-azstorageblobcontent.
The errors seem to be things like "Couldn't connect" or "Retry count exceeded" etc. I am unable to use the -debug switch as it says its interactive.
I have tested by including the access token inside the script, eliminating jenkins from handling the secrets but i get the same issue.
Jenkins is running the latest version, latest PowerShell plugin and latest version of java. it is also calling the 64 bit powershell session as expected.
I am also aware that there is a Jenkins Blob storage plugin however due to the amount of additional work that is required, it makes a lot more sense i use the PowerShell Modules inside a PowerShell Script to run this.
I would really appreciate if anyone has any ideas about this please, It has been driving me nuts for weeks.
Many Thanks

Related

Difference between Azure CLI and PowerShell

I am asking this question as Windows user but I request you not to limit the answer for windows only.
I tried executing Azure CLI commands in Power-Shell and they are executing successfully but not the other way around. In that case why do we have 2 separate command sets? Why not just work on Power-Shell? I have only tried some basic commands and they are all working except filter commands | find only works on CLI and | Select only works in Power-Shell.
I know that Azure CLI is for cross platform support. But is there any difference for Windows user? Are there any consequences to run CLI command on Power-Shell?
Thanks in advance.
Here are my opinions from using both. In no way am I saying one is better than the other. They both have their pros and cons.
Azure CLI is cross-platform command-line tool for managing Azure resources, and it can run in Windows, Mac and Linux. This also means it can run on Windows PowerShell. Its more flexible than Azure PowerShell since its a binary and can run inside any OS default shell.
Are there any consequences to run CLI command on Power-Shell?
Updating can be a bit of a pain. If you want to update it on Windows you have to re-install the MSI with the instructions from Install Azure CLI on Windows. Updating is easier on other platforms, and makes me only want to only use Azure PowerShell on Windows PowerShell. A work around is to use WSL on Windows, then you can run Azure CLI on Linux inside a Windows machine. You can install WSL at Windows Subsystem for Linux Installation Guide for Windows 10. I find updating the azure-cli package much easier on Linux using apt-get than the Windows equivalent. You can have a look at Install Azure CLI with apt on how to install the Azure CLI package on Linux.
Another difference is that you have to chain multiple commands with Azure CLI, such as az group list vs Get-AzResourceGroup from Azure PowerShell. You also can't run Get-Help with Azure CLI commands like you do with Azure PowerShell, which is a huge game changer for me, since I find the PowerShell help system to be very helpful displaying in-depth information about PowerShell Cmdlets. The Azure CLI help info is found with az --help, but is not as comprehensive as Get-Help.
Tab completion also doesn't work in Azure CLI when using Windows PowerShell. This makes typing a bit of a pain as well, another reason I use WSL inside of Windows. You can have a look at the other alternatives at this Autocompletion support in Windows command line GitHub issue.
Azure PowerShell on the other hand is a set of PowerShell Cmdlets for managing Azure resources from the command-line, and only works in Windows PowerShell and PowerShell Core. This also means that if another OS such as Mac or Linux is running PowerShell Core, then it can run Azure PowerShell as well.
I have only tried some basic commands and they are all working except filter commands | find only works on CLI and | Select only works in Power-Shell.
Select-Object or the shorthand Select is a PowerShell Cmdlet, so it only works on PowerShell objects. find can search a string or text file. Furthermore, if you are searching for data from Azure CLI, you should use the --query parameter instead of find, since find is limited to searching strings inside text. Azure CLI uses the JMESPath query language to search for data inside the JSON output you receive. If your comfortable with this query language then searching for data using Azure CLI shouldn't be too much of an issue. Additionally, you can also use Azure CLI commands inside PowerShell scripts, but not vice-versa.
Summary
If you deal with multiple platforms or want to write scripts with others that use different platforms, using Azure CLI is a good choice. However, if you mainly deal with Windows systems and work with others who do so as well, then using Azure PowerShell is a good idea. If your like me and have to use different platforms, then installing both is a good idea. If you still just want to use Azure PowerShell on different OS platforms, then you need PowerShell Core.
For simple tasks, like quickly looking up resources in cloud shell or writing quick scripts, Azure CLI is good to use and less verbose than Azure PowerShell. If you already use bash a lot, this will feel more natural, and adding Azure CLI commands to existing scripts will be a simple task. As others have also said, there is nothing stopping you from adding Azure CLI commands to powershell scripts, which allows you to deserialize the JSON output using ConvertFrom-Json into a PSCustomObject.
For more complex tasks, Azure PowerShell is preferable, since working with .NET objects/OOP principals is much easier than parsing the JSON text given from Azure CLI. This is one reason I try to use Azure PowerShell when I can.
Azure CLI does benefit from being idempotent, so running the same command against the resources won't require any null checking like in Azure PowerShell. If this becomes an issue, then you can run ARM templates in Azure PowerShell, which are idempotent.
Update
As #AimusSage helpfully pointed out in the comments, PowerShell 7.0 has recently been released, replacing the name PowerShell Core or PowerShell 6.x. You can read more at Announcing PowerShell 7.0.
Another Idea
If you want to maintain OOP principals from Azure PowerShell, but use something that is easier for Linux Sys Admins to use, then you can consider using the Azure SDK for Python. I have used this in the past when I wanted to run scripts in a Linux host, but didn't want to use Azure CLI or install PowerShell.
I like the previous answers, I just want to add a different point of view for people in the enterprise world that is forced to pick one:
In that case why do we have 2 separate command sets? Why not just work on Power-Shell?
Rephrasing: Both Az CLI and Az PowerShell just call the same set of APIs, the Azure APIs.
This is important because theoretically and eventually you will be able to do everything in both ways
So why did Microsoft creates and maintains two ways of doing the same thing?
Martin Fowler ones said: but remember, the skill of the team will outweigh any monolith/microservice choice
If you change the monolith/microservice by PowerShell/Bash then this answers the question.
I believe that there are people with decades of development of systems using Bash, and there are other teams that are heavy users of PowerShell. Microsoft does not want them to have to learn a whole new programming language to be able to use Azure.
Summary:
If your team is familiar with PowerShell, go with PowerShell and do as much native PowerShell stuff as possible. This way you benefit from things like error handling, OOP concepts,environment settings, parallelization, etc
If yout team are linux admins, heavy users of jenkins, with million lines of Bash to automate other things already there, and working with Bash for their entire life, go with CLI and keep consistency across all the already built tools
I've noticed when doing the MS Azure training (AZ-900 fundamentals and AZ-303 Azure Architect) is that the exercises are done in Azure CLI.
That's not to say it's better, but if you're wanting to do the exams it might be worth being familiar with it. For the record, I'm a PowerShell guy.
the existing answer is silly way of looking at this question. and misleading.
The biggest difference is that azure cli is a binary (that can run on different platforms) and Powershell is a shell that works across platforms. Azure Powershell is a bunch of Powershell modules, everything else derives from that.
find cannot work in the cli, because there is no in the cli, because its not a shell. find works perfectly fine in Powershell on Windows, because its a binary in the Windows OS, whereas select is a cmdlet in Powershell and hence it will not work in command line on Windows (or bash on Linux).
Furthermore, if you are searching for data from Azure CLI, you should
use the --query parameter instead of find
this is also debatable, JMESPath query language is overcomplicated for no particular reason and I dont know how Azure Cli is actually compatible with the official JMESPath documentation. I prefer to use Powershell to run Azure Cli commands and just parse output JSON with Powershell. Obviously, you might not be as comfortable with Powershell as I am and you might not find this convenient.
Another issue with Azure Powershell that does not seem to be the case with azure cli is the different versions. I have spent days figuring out which version of which command for which script in my pipeline needs to be what. Seriously the most ridiculous assinined

Can't bind parameter 'log' to type TraceWriter when running Azure function template locally

I met an issue when using Azure function.
I create a Azure Functions v1(.Net Framework) Http Trigger template in VS. But when I try to run it directly, exceptions thrown.
I am using latest version of VS(15.6.7) and Azure Functions and Web Job tools(15.0.40502.0).
When I run it first time, no prompt for installation of anything, it runs on 1.0.10 Azure Function CLI. After I restart my VS and try to run it again, VS asks to download 1.0.12.1 Function CLI. The download seems failed as I still see 1.0.10 on the tile of window. The exception remains all the time.
Any idea? It doesn't make sense since I just try to run a template.
The root cause is shown in your screenshot.
Starting Host(...,Version=1.0.11232.0)
It means your function Cli is 1.0.4 other than 1.0.10 actually, here's release note of 1.0.4. After tests, find that old version does cause this exception. This version issue roots in failing to download latest Azure Function Core Tools trough VS.
Update for official solution
Handle downloading problem of Azure Function Core Tools. Try the first option to download using VS again, if it still fails, use the second option, PS scripts.
Below is the original solution, similar to the first and third official option, just ignore it
More Details
Azure Functions and Web Jobs Tools is updated to 15.0.40502.0 recently and mechanism of using local function Cli is also changed. Release Notes.
Tools now consume a feed which keeps templates, build tools, and the runtime up to date whenever there is a change made in the service.
The first time we create an Azure function after this update, we can see the tip on the dialog Making sure all templates are up to date.... It means VS is downloading necessary cli and templates to this folder C:\Users\UserName\AppData\Local\AzureFunctionsTools.
After a while, we can see the tip changes as
If we don't wait the downloading to complete and create project directly, it will prompt that VS is downloading 1.0.12.1 Cli. And everything should work fine after that.
As for your problem, I reproduce it once due to slow network. It fails to download those files and tries to use old version cli downloaded by VS before.
Solution
I recommend you to delete AzureFunctionsTools folder and restart your VS to download it again.
If it still fails to download, you can download it manually.
You can find download link of cli, itemTemplates and projectTemplates in C:\Users\UserName\AppData\Local\AzureFunctionsTools\feed.json.
And the folder structure in C:\Users\UserName\AppData\Local\AzureFunctionsTools\Releases\1.0.12.1 is like this
cli
--func.exe
....
templates
--ItemTemplates.nupkg
--ProjectTemplates.nupkg
manifest.json
Content of manifest.json
{
"ReleaseName": "1.0.12.1",
"CliEntrypointPath":"C:\\Users\\UserName\\AppData\\Local\\AzureFunctionsTools\\Releases\\1.0.12.1\\cli\\func.exe",
"TemplatesDirectory": "C:\\Users\\UserName\\AppData\\Local\\AzureFunctionsTools\\Releases\\1.0.12.1\\templates",
"FunctionsExtensionVersion": "~1",
"SdkPackageVersion": "1.0.13"
}

Release Failure from TFS CI

I am trying to deploy an application to the Azure Service Fabric using the release definition task. When it gets to the task to deploy the server is returning the following error:
The type initializer for 'Microsoft.ServiceFabric.Powershell.Constants' threw an exception
I checked the Endpoint configuration and it appears to be set up as it is supposed to be:
No Authentication (this is an internal text box)
Cluster endpoint: tcp://[service fabric server]:19000
It downloads the artifacts without a problem, but in deploy it searches for the paths for publish profile and application package and finds them. After it finds them it throws the error. I have tried replacing TCP in the endpoint with http, added and removed the :19000 as well and all I get is this error. I have been searching online with little success. Any help to this end is much appreciated.
John
After lots of researching trying every suggestion I could find, we decided try and connect to the machine via Powershell on the box and it too was returning this error. So we uninstalled the SDK and re-installed it and the connection could be made and the builds started to work. I don't know exactly why it failed, but apparently a re-install did the trick. It may have been a bad install, or it could have been a versioning problem. Either way, try a re-install first.

Azure Powershell Scripts Fail Siliently

I've been working with the Azure Powershell scripts from Microsoft over the last new days and am becoming frustrated with how often they seem to fail silently or give very misleading error messages. I have tried using the -verbose option, but this isn't very verbose.
Does any have any tips of things I can try to beef up the feedback when something goes wrong?
For example: Save-AzureServiceProjectPackage is reporting that the cloud_package.cspkg is created but no file is created on disk. If I delete my node_modules folder it works correctly and reports the same thing!
I've raised this direct with the team, but it appears the problem is with cspack.exe which the powershell calls: https://github.com/Azure/azure-sdk-tools/issues/2689
I had similar problems.
What didn't work: to install the latest SDK for Azure (Libraries for .NET and Azure Powershell)
What did work: to shorten the longest directory path.
For instance, in your deepest subdirectories, you delete everything not needed. Or move your $DEV_ROOT directory to the drive's top level. It's not a nice solution though.

PowerShell Script in TFS Build not uploading to Azure

After searching on Google, I've managed to put together a powershell script based on this site to upload my azure project. When I try to add it to TFS build process using the steps on the same site I get the following output:
My build says it's been running for 7 hours and if I login to windows azure there is no mention of the package. I have run it numerous other times since but not for as long, even after 10 minutes it still hasn't changed anything on the Azure dashboard.
If I run the same command from a command prompt on the build server then it runs and uploads the staging environment and it takes about 8-10 minutes to complete. Is there anything special I have to do to get it to run from TFS Build, or is there a way that I can get more meaningful output?
Finally found it. The default.publishsettings didn't have permission for the build service to read it. Added full permissions for the service and it now works.

Resources