I am confused.
Can you install HDInsight Server (multiple nodes) locally, NOT on Azure?
I don't want to install Hortonworks (I know that HDInsight uses their distribution).
Also, I don't want to install any kind of emulator or developer preview with one node.
Again, I would like to install HDInsight on my windows servers on PRODUCTION, how can I do that?
Chicago, I don't think this is possible. HDInsight is a service offered as part of the public Azure offering. If you want to install multiple nodes locally you will need to go with Hortonworks or the Apache Hadoop distribution.
Related
I created Windows platform HDInsight Cluster in azure. And then logged into Cluster by using remote desktop credentials after that I am trying to install data management gateway.
I followed the below link to install data management gateway for connecting HDFS: Move data between on-premises sources and the cloud with Data Management Gateway
Whenever I run the msi file in HD Insight Cluster I got the error like “the system administrator has set policies to prevent this installation”. For that I followed this link: The System Administrator has set policies to prevent this installation but I am able to set the reg key under installer even I run the regedit in “Run as Administrator”.
So, how can I install data management gate way in HDInsight Cluster?
I'm using DC/OS 1.8 when i installed Cassandra Service from the DC/OS Universe.
If i stop & start my cluster (master and all nodes) the service doesn't start and i have to uninstall the service, delete all file in the agents node and after install Cassandra service.
P.S. My Cassandra Cluster is installed on Azure with Azure Container Service.
Do you have any idea?
Few questions before we even go further investigation.
1. Have you seen tried to reproduce the issue ? How it is like ?
2. Have you explored DC/OS 1.8 without Azure ? Maybe it is DC/OS issue.
In general, we (Azure Container Service) will have the issue if DC/OS itself is buggy. It is always worth to mine the known issues on DC/OS github first. And yes, we need log
I'm logged into the HDInsight remote desktop machine and I'm on the Hadoop Console. But when I type Hive I get command not recognized although it works on my developer preview. Have I not configured my HDinsight cluster correctly. Any pointers would be appreciated.
From the Hadoop command prompt, type the following command:
cd %hive_home%\bin
It is recommended to submit hive job using Windows Azure PowerShell or HDInsight .NET SDK. For more information, see http://www.windowsazure.com/en-us/manage/services/hdinsight/submit-hadoop-jobs-programmatically/.
I'm trying to play with Hadoop on Azure using HDInsight, but am a bit confused on how to run a Hadoop job on my newly created cluster. So far I've created an HDInsight cluster and attached a Storage Account to it. I also have the azure-cli installed on my local OS X box.
There's an Azure tutorial on launching Hadoop jobs, but it uses PowerShell, which I don't think is available via the Azure cross-platform CLI.
Aside form starting up a Windows VirtualBox, can a job get created via the Azure Web UI (e.g. like Amazon EMR provides) or via some other command line arguments that would be compatible with OS X?
Thanks
Login using Remote Desktop Protocol into the head node of the cluster and use CLI tools there.
Or submit jobs using WebHCatalog REST APIs (e.g. using curl).
HDInsight currently has XPlat CLI tools for cluster provisioning. Job submission XPlat CLI tools will be available later.
I am able to create a new Hadoop cluster through the interface, but need to create a new cluster on request. Does anyone know if an API exists to create a new cluster?
Not yet. As of now (Preview) you must be using Windows Azure Management Portal Interface to create Hadoop cluster in your windows Azure Subscription.
As most of the Windows Azure Management functionalities are available on Powershell, it is possible to have such functionality built into Powershell over REST as described here however I don't know any immediate plans.
Yes, you can.
The Azure CLI allows you to control HDInsight clusters from a batch file for example. You then get a bunch of HDInsight control functions. Type
azure hdinsight
to see all the built in help. It covers all the basics (listing, creating configuring clusters) and the multiple storage account functionality.
This is I believe based on the nodejs sdk. To get going, install nodejs and
npm install azure-cli
This should give you what you need to be able to manage the clusters from the command line.
By asking "Create a new Hadoop cluster", I believe you mean Hadoop HDInsight Cluster.
If yes, then we can create a powershell(.ps1) script, which can do the job for you.
Here is the sample script which might be useful.
http://mydailyfindingsit.blogspot.in/2016/01/create-script-hdinsight-cluster.html