I'm trying to connect to an azure data explorer but I keep getting a non descriptive error. I'm following this tutorial.
https://learn.microsoft.com/en-us/sql/azure-data-studio/notebooks/notebooks-kqlmagic?view=sql-server-ver16.
Has anyone seen this?
click here for screenshot
I was trying to connect to azure data explorer from Azure Machine Learning Studio notebooks. I also tried it in Jupyter notebooks with an anaconda environment and I got the same error.
However, the command %reload_ext Kqlmagic worked for me
Maybe its because that Azure login has multiple directories?
I am trying to use databricks CLI and invoke the databricks configure. At the point where we need to enter the token,I am neither able to type anything or copy paste anything.
You need to just copy the token and paste and hit enter.
Note: Even after pasting the token it will show as blank.
.
Now you can run the databricks fs ls command to check whether you have successfully configured.
For more details, refer similar thread: databricks configure using cmd and R
With databricks-connect we can successfully run codes written in Databricks or Databricks notebook from many IDE. Databricks has also created many magic commands to support their feature with regards to running multi-language support in each cell by adding commands like %sql or %md. One issue I am facing currently is when I try to execute Databricks notebooks in Pycharm is as follows:
How to execute Databricks specific magic command from PyCharm.
E.g.
Importing a script or notebook in Done in Databricks using this command-
%run
'./FILE_TO_IMPORT'
Where as in IDE from FILE_TO_IMPORT import XYZ works.
Again everytime I download Databricks notebook it comments out the magic commands and that makes it impossible to be used anywhere outside Databricks environment.
It's really inefficient to convert all databricks magic command everytime I want to do any developement.
Is there any configuration I could set which automatically detects Databricks specific magic commands?
Any solution to this will be helpful. Thanks in Advance!!!
Unfortunately, as per the databricks-connect version 6.2.0-
" We cannot use magic command outside the databricks environment directly. This will either require creating custom functions but again that will only work for Jupyter not PyCharm"
Again, since importing py files requires %run magic command so this also becomes a major issue. A solution to this is by converting the set of files to be imported as a python package and add it to the cluster via Databricks UI and then import and use it in PyCharm. But this is a very tedious process.
I am asking this question as Windows user but I request you not to limit the answer for windows only.
I tried executing Azure CLI commands in Power-Shell and they are executing successfully but not the other way around. In that case why do we have 2 separate command sets? Why not just work on Power-Shell? I have only tried some basic commands and they are all working except filter commands | find only works on CLI and | Select only works in Power-Shell.
I know that Azure CLI is for cross platform support. But is there any difference for Windows user? Are there any consequences to run CLI command on Power-Shell?
Thanks in advance.
Here are my opinions from using both. In no way am I saying one is better than the other. They both have their pros and cons.
Azure CLI is cross-platform command-line tool for managing Azure resources, and it can run in Windows, Mac and Linux. This also means it can run on Windows PowerShell. Its more flexible than Azure PowerShell since its a binary and can run inside any OS default shell.
Are there any consequences to run CLI command on Power-Shell?
Updating can be a bit of a pain. If you want to update it on Windows you have to re-install the MSI with the instructions from Install Azure CLI on Windows. Updating is easier on other platforms, and makes me only want to only use Azure PowerShell on Windows PowerShell. A work around is to use WSL on Windows, then you can run Azure CLI on Linux inside a Windows machine. You can install WSL at Windows Subsystem for Linux Installation Guide for Windows 10. I find updating the azure-cli package much easier on Linux using apt-get than the Windows equivalent. You can have a look at Install Azure CLI with apt on how to install the Azure CLI package on Linux.
Another difference is that you have to chain multiple commands with Azure CLI, such as az group list vs Get-AzResourceGroup from Azure PowerShell. You also can't run Get-Help with Azure CLI commands like you do with Azure PowerShell, which is a huge game changer for me, since I find the PowerShell help system to be very helpful displaying in-depth information about PowerShell Cmdlets. The Azure CLI help info is found with az --help, but is not as comprehensive as Get-Help.
Tab completion also doesn't work in Azure CLI when using Windows PowerShell. This makes typing a bit of a pain as well, another reason I use WSL inside of Windows. You can have a look at the other alternatives at this Autocompletion support in Windows command line GitHub issue.
Azure PowerShell on the other hand is a set of PowerShell Cmdlets for managing Azure resources from the command-line, and only works in Windows PowerShell and PowerShell Core. This also means that if another OS such as Mac or Linux is running PowerShell Core, then it can run Azure PowerShell as well.
I have only tried some basic commands and they are all working except filter commands | find only works on CLI and | Select only works in Power-Shell.
Select-Object or the shorthand Select is a PowerShell Cmdlet, so it only works on PowerShell objects. find can search a string or text file. Furthermore, if you are searching for data from Azure CLI, you should use the --query parameter instead of find, since find is limited to searching strings inside text. Azure CLI uses the JMESPath query language to search for data inside the JSON output you receive. If your comfortable with this query language then searching for data using Azure CLI shouldn't be too much of an issue. Additionally, you can also use Azure CLI commands inside PowerShell scripts, but not vice-versa.
Summary
If you deal with multiple platforms or want to write scripts with others that use different platforms, using Azure CLI is a good choice. However, if you mainly deal with Windows systems and work with others who do so as well, then using Azure PowerShell is a good idea. If your like me and have to use different platforms, then installing both is a good idea. If you still just want to use Azure PowerShell on different OS platforms, then you need PowerShell Core.
For simple tasks, like quickly looking up resources in cloud shell or writing quick scripts, Azure CLI is good to use and less verbose than Azure PowerShell. If you already use bash a lot, this will feel more natural, and adding Azure CLI commands to existing scripts will be a simple task. As others have also said, there is nothing stopping you from adding Azure CLI commands to powershell scripts, which allows you to deserialize the JSON output using ConvertFrom-Json into a PSCustomObject.
For more complex tasks, Azure PowerShell is preferable, since working with .NET objects/OOP principals is much easier than parsing the JSON text given from Azure CLI. This is one reason I try to use Azure PowerShell when I can.
Azure CLI does benefit from being idempotent, so running the same command against the resources won't require any null checking like in Azure PowerShell. If this becomes an issue, then you can run ARM templates in Azure PowerShell, which are idempotent.
Update
As #AimusSage helpfully pointed out in the comments, PowerShell 7.0 has recently been released, replacing the name PowerShell Core or PowerShell 6.x. You can read more at Announcing PowerShell 7.0.
Another Idea
If you want to maintain OOP principals from Azure PowerShell, but use something that is easier for Linux Sys Admins to use, then you can consider using the Azure SDK for Python. I have used this in the past when I wanted to run scripts in a Linux host, but didn't want to use Azure CLI or install PowerShell.
I like the previous answers, I just want to add a different point of view for people in the enterprise world that is forced to pick one:
In that case why do we have 2 separate command sets? Why not just work on Power-Shell?
Rephrasing: Both Az CLI and Az PowerShell just call the same set of APIs, the Azure APIs.
This is important because theoretically and eventually you will be able to do everything in both ways
So why did Microsoft creates and maintains two ways of doing the same thing?
Martin Fowler ones said: but remember, the skill of the team will outweigh any monolith/microservice choice
If you change the monolith/microservice by PowerShell/Bash then this answers the question.
I believe that there are people with decades of development of systems using Bash, and there are other teams that are heavy users of PowerShell. Microsoft does not want them to have to learn a whole new programming language to be able to use Azure.
Summary:
If your team is familiar with PowerShell, go with PowerShell and do as much native PowerShell stuff as possible. This way you benefit from things like error handling, OOP concepts,environment settings, parallelization, etc
If yout team are linux admins, heavy users of jenkins, with million lines of Bash to automate other things already there, and working with Bash for their entire life, go with CLI and keep consistency across all the already built tools
I've noticed when doing the MS Azure training (AZ-900 fundamentals and AZ-303 Azure Architect) is that the exercises are done in Azure CLI.
That's not to say it's better, but if you're wanting to do the exams it might be worth being familiar with it. For the record, I'm a PowerShell guy.
the existing answer is silly way of looking at this question. and misleading.
The biggest difference is that azure cli is a binary (that can run on different platforms) and Powershell is a shell that works across platforms. Azure Powershell is a bunch of Powershell modules, everything else derives from that.
find cannot work in the cli, because there is no in the cli, because its not a shell. find works perfectly fine in Powershell on Windows, because its a binary in the Windows OS, whereas select is a cmdlet in Powershell and hence it will not work in command line on Windows (or bash on Linux).
Furthermore, if you are searching for data from Azure CLI, you should
use the --query parameter instead of find
this is also debatable, JMESPath query language is overcomplicated for no particular reason and I dont know how Azure Cli is actually compatible with the official JMESPath documentation. I prefer to use Powershell to run Azure Cli commands and just parse output JSON with Powershell. Obviously, you might not be as comfortable with Powershell as I am and you might not find this convenient.
Another issue with Azure Powershell that does not seem to be the case with azure cli is the different versions. I have spent days figuring out which version of which command for which script in my pipeline needs to be what. Seriously the most ridiculous assinined
When trying to reference/load a dsource or dprep file generated with a data source file from blob storage, I receive the error "No files for given path(s)".
Tested with .py and .ipynb files. Here's the code:
# Use the Azure Machine Learning data source package
from azureml.dataprep import datasource
df = datasource.load_datasource('POS.dsource') #Error generated here
# Remove this line and add code that uses the DataFrame
df.head(10)
Please let me know what other information would be helpful. Thanks!
Encountered the same issue and it took some research to figure out!
Currently, data source files from blob storage are only supported for two cluster types: Azure HDInsight PySpark and Docker (Linux VM) PySpark
In order to get this to work, it's necessary to follow instructions in Configuring Azure Machine Learning Experimentation Service.
I also ran az ml experiment prepare -c <compute_name> to install all dependencies on the cluster before submitting the first command, since that deployment takes quite a bit of time (at least 10 minutes for my D12 v2 cluster.)
Got the .py files to run with HDInsight PySpark compute cluster (for data stored in Azure blobs.) But .ipynb files are still not working on my local Jupyter server - the cells never finish.
I'm from the Azure Machine Learning team - sorry you are having issues with Jupyter notebook. Have you tried running the notebook from the CLI? If you run from the CLI you should see the stderr/stdout. The IFrame in WB swallows the actual error messages. This might help you troubleshoot.