I just created a free account to Azure and created a hadoop cluster on HDInsight. However, it dose not in anyway show how to launch a Pig client and create pig latin job and run it?
Unfortunately, you cannot use Ambari UI to run pig latin jobs.
Note: To process data using Pig, will need to open an SSH console that is connected to your cluster and then run the pig latin using local mode or mapreduce mode:
If you are using a Windows client computer:
In the Microsoft Azure portal, on the HDInsight Cluster blade for your HDInsight cluster, click Secure Shell, and then in the Secure Shell blade, in the Hostname list, note the Host name for your cluster (which should be your_cluster_name-ssh.azurehdinsight.net).
Open PuTTY, and in the Session page, enter the host name into the Host Name box. Then under Connection type, select SSH and click Open. If a security warning that the host certificate cannot be verified is displayed, click Yes to continue.
When prompted, enter the SSH username and password you specified when provisioning the cluster (not the cluster login username).
If you are using a Mac OS X or Linux client computer:
In the Microsoft Azure portal, on the HDInsight Cluster blade for your HDInsight cluster, click Secure Shell, and then in the Secure Shell blade, in the Hostname list, select the hostname for your cluster. then copy the ssh command that is displayed, which should resemble the following command – you will use this to connect to the head node.
ssh sshuser#your_cluster_name-ssh.azurehdinsight.net
Open a new terminal session, and paste the ssh command, specifying your SSH user name (not the cluster login username).
If you are prompted to connect even though the certificate can’t be verified, enter yes.
When prompted, enter the password for the SSH username.
Once you connected to your cluster, to run pig latin as shown:
You can execute Pig Latin statements:
Using grunt shell or command line
In mapreduce mode or local mode
Either interactively or in batch
Reference: Pig Manual
Related
I'm looking to connect VS Code to Azure Cloudshell, especially for the file explorer component.
It is a bit of an anti-pattern, but cloudshell is the only way in which I am able to run some commands that connect to Azure resources (spinning up a VM is not allowed).
I know the 'remote-ssh' extension can connect to a remote machine and provide a nice file explorer, however, for cloudshell I do not have ssh available. However, there is a connection possible through the 'Azure account' extension.
(There is also a 'remote tunnel' extension that is able to establish connections over different protocol than ssh).
This is the connection that 'Azure account' sets up with the cloudshell:
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
tcp ESTAB 0 0 10.xxx.xx.169:http-alt 10.xxx.xx.29:57404
tcp ESTAB 0 0 10.xxx.xx.169:51710 20.xxx.xx.200:microsoft-ds
Does anybody know if this can be used to get a full remote development experience in VScode connected to Azure cloudshell?
(I tried to install the vs code server rpm package in cloudshell, failed due to lacking yum and sudo permissions.)
Answer:
TLDR: type code . in azure Cloudshell in browser to get file editor.
(Using own IDE is not possible, only cloudshell as terminal)
Check the below steps to connect to Azure Cloud shell (PowerShell/Bash).
Open VSCode, = >Extensions => search for Azure Account.
Check whether Azure Account is Installed and Enabled.
If it is not done, Install and Enable it.
We need to Login to Azure Account to Open the Cloud shell.
In View tab => Command Palette => Azure: Sign in
OR
Press Ctrl + Shift + P => Azure: Sign in.
Click on Sign in and login to Azure.
You
In the right, click on the dropdown symbol beside + .You will find an option for Azure Cloud shell (PowerShell / Bash).
Select Azure Cloud shell (PowerShell) .
If it is not sign into Azure, you will get an option to Sign in.
Once after Sign, you will get the below message in the terminal.
To make sure we are connected to Azure Cloud shell, execute any of the PowerShell command and check once.
file explorer component
Thanks #Brendan Burn for the Post.
Refer this SO Thread regarding File Explorer.
I've created a VM of Apache Solr by Bitnami on Azure, specifying my username / password in the creation process .
When i run : ssh my_username#my_vm_ip it works
But in the web portal http://my_vm_ip/solr , a popup username/password appears . I use same credentials than SSH and it fails
Any idea?
The application password generated by the Bitnami VM is generated uniquely for each VM, and is shown on the application vm console:
I'm not sure how you'd display the console login screen on Azure, but you can check the boot diagnostics log at least. The banner message is usually stored either in /etc/issue.* or /etc/motd, so I'd check that as well.
The default user name is user.
The default password you need to ssh to your VM and check /var/log/syslog. Search Bitnami, you will get like below:
More information about this, you could check this FAQ.
I created a HDinsight Cluster and was trying to log in to Shell using Putty
<Clustername>-ssh.azurehdinsight.net using the given credentials while creating the Cluster.
Having problems logging using Putty after I gave credentials it's showing access denied.
Are there any other permissions that I need to provide to log in using Putty?
At the time of creating the cluster, you need to select the SSH option and specify the public key. This can then be used to submit jobs by logging in to the head node. I had blogged about it recently
https://nileshgule.blogspot.sg/2017/10/submit-apache-spark-job-from-command.html
Hope this helps
No need any permissions that need to provided to log in using putty.
If you are using the following syntax to connect using putty(you will receive Access denied):
ssh sshuser#clustername-ssh.azurehdinsight.net
Use the following to connect using putty:
sshuser#clustername-ssh.azurehdinsight.net
You can securely connect to the below endpoints in the HDInsight cluster with an SSH client. you may refer Documentation.
You can also try the below method to connect HDInsight cluster using PuTTY.
Goto Configuration -> SSH + Cluster login -> copy the cluster name without ssh -> enter the details in PuTTy and click connect -> enter the password to connect.
Refer the below screenshots for more details.
How do I access pig log files on HDInsight?
When pig errors the output says:
Details at logfile: C:\apps\dist\hadoop-1.2.0.1.3.7.1-01293\logs\pig_1399635949926.log
I can't find anything like that in the blob store and I have no idea how to access that path.
Cheers.
It is because it exists on the C:\ drive of the headnode. Simple steps to access it:
Go the manage.windowsazure.com and go to your HDI cluster
Enable remote access, choose a user and a password (note admin is reserved)
Then just login the machine
Open the explorer and go to that path
Either copy&paste to your local machine or open it in notepad
It's really big problem for me - ugly UI, where I can't see ssh connection requisites for my new virtual machine instance. Is it possible to change password? Password and login from my manage.windowsazure.com panel is not approach.
I guess you are talking about a Linux VM? When creating a new VM you have to supply a username and password or you can upload an SSH key. The standard username is azureuser. There is no way to reset the password of a VM w/o having access to it.
Update: By now it's possible to create a new User/Password or upload a new SSH key via the new Azure Portal.
Got to https://portal.azure.com
Select your virtual machine
Click on Settings
Select Reset Password
Generate new certificate using this article: http://azure.microsoft.com/en-us/documentation/articles/linux-use-ssh-key/
In Azure management console, go to "CLOUD SERVICES" -> your service -> certificates
Upload you new generated certificate
Now you can login with ssh -i ~/.ssh/[private_key] azureuser#[yourappname].cloudapp.net
Have you tried changing it on the machine.
Login to the Virtual Machine via SSH with your current password. and run chpwd