I'm logged into the HDInsight remote desktop machine and I'm on the Hadoop Console. But when I type Hive I get command not recognized although it works on my developer preview. Have I not configured my HDinsight cluster correctly. Any pointers would be appreciated.
From the Hadoop command prompt, type the following command:
cd %hive_home%\bin
It is recommended to submit hive job using Windows Azure PowerShell or HDInsight .NET SDK. For more information, see http://www.windowsazure.com/en-us/manage/services/hdinsight/submit-hadoop-jobs-programmatically/.
Related
I have set up the HDI 3.6 spark 2.1.0 HDInsight cluster.
Now I need to submit a spark job using oozie workflow,for doing this I need to configure the workflow.xml using Ambari.
The problem arises when accessing oozie webUI from Ambari as below:
while click on Oozie web UI I am getting the following error for loading the url:
https://samplecluster.azurehdinsight.net/da/host/hn0-cloudo.fabtnv3ti3juvluair5dbgfudc.hx.internal.cloudapp.net/port/11000/oozie?user.name=sathyaraj
error:
{
"status": 404,
"message": "Not Found"
}
To access the Oozie Web UI, use the following steps:
Create an SSH tunnel to the HDInsight cluster. For information, see the Use SSH Tunneling with HDInsight document.
Once a tunnel has been created, open the Ambari web UI in your web browser. The URI for the Ambari site is https://CLUSTERNAME.azurehdinsight.net. Replace CLUSTERNAME with the name of your Linux-based HDInsight cluster.
From the left side of the page, select Oozie, then Quick Links, and finally Oozie Web UI.
For more details, refer “Oozie Web UI”.
I created Windows platform HDInsight Cluster in azure. And then logged into Cluster by using remote desktop credentials after that I am trying to install data management gateway.
I followed the below link to install data management gateway for connecting HDFS: Move data between on-premises sources and the cloud with Data Management Gateway
Whenever I run the msi file in HD Insight Cluster I got the error like “the system administrator has set policies to prevent this installation”. For that I followed this link: The System Administrator has set policies to prevent this installation but I am able to set the reg key under installer even I run the regedit in “Run as Administrator”.
So, how can I install data management gate way in HDInsight Cluster?
I am confused.
Can you install HDInsight Server (multiple nodes) locally, NOT on Azure?
I don't want to install Hortonworks (I know that HDInsight uses their distribution).
Also, I don't want to install any kind of emulator or developer preview with one node.
Again, I would like to install HDInsight on my windows servers on PRODUCTION, how can I do that?
Chicago, I don't think this is possible. HDInsight is a service offered as part of the public Azure offering. If you want to install multiple nodes locally you will need to go with Hortonworks or the Apache Hadoop distribution.
I'm trying to play with Hadoop on Azure using HDInsight, but am a bit confused on how to run a Hadoop job on my newly created cluster. So far I've created an HDInsight cluster and attached a Storage Account to it. I also have the azure-cli installed on my local OS X box.
There's an Azure tutorial on launching Hadoop jobs, but it uses PowerShell, which I don't think is available via the Azure cross-platform CLI.
Aside form starting up a Windows VirtualBox, can a job get created via the Azure Web UI (e.g. like Amazon EMR provides) or via some other command line arguments that would be compatible with OS X?
Thanks
Login using Remote Desktop Protocol into the head node of the cluster and use CLI tools there.
Or submit jobs using WebHCatalog REST APIs (e.g. using curl).
HDInsight currently has XPlat CLI tools for cluster provisioning. Job submission XPlat CLI tools will be available later.
I am able to create a new Hadoop cluster through the interface, but need to create a new cluster on request. Does anyone know if an API exists to create a new cluster?
Not yet. As of now (Preview) you must be using Windows Azure Management Portal Interface to create Hadoop cluster in your windows Azure Subscription.
As most of the Windows Azure Management functionalities are available on Powershell, it is possible to have such functionality built into Powershell over REST as described here however I don't know any immediate plans.
Yes, you can.
The Azure CLI allows you to control HDInsight clusters from a batch file for example. You then get a bunch of HDInsight control functions. Type
azure hdinsight
to see all the built in help. It covers all the basics (listing, creating configuring clusters) and the multiple storage account functionality.
This is I believe based on the nodejs sdk. To get going, install nodejs and
npm install azure-cli
This should give you what you need to be able to manage the clusters from the command line.
By asking "Create a new Hadoop cluster", I believe you mean Hadoop HDInsight Cluster.
If yes, then we can create a powershell(.ps1) script, which can do the job for you.
Here is the sample script which might be useful.
http://mydailyfindingsit.blogspot.in/2016/01/create-script-hdinsight-cluster.html