I have trouble running MPI code on cluster and use the following command to run the code and tried it with ip addresses as well.
mpiuser1#comp-asdf-4:~$ mpirun -np 4 -hosts master, slave1 ./myexe
and have following error
mpirun: Error: unknown option "-o"
Type 'mpirun --help' for usage.
i have a cluster of 2 nodes one master and slave and I follow this tutorial http://mpitutorial.com/tutorials/running-an-mpi-cluster-within-a-lan/ to create a cluster. I am using ubuntu (16.04.1), gcc version 5.4.0, openmpi version 3.0.1. The same error reported by other user but he wasn't using the same version of mpi on all cluster nodes and have tried the suggesstions posted here Error while running MPI cluster program in LAN but problem is still there.
Specifying hosts with OpenMPI works with --host. This is a bit different than with MPICH2 used in the tutorial you follow.
$ mpirun -np 4 --host master, slave1 ./myexe
I recommend you pay close attention to Open MPI man pages and FAQ.
mpirun(1) man page (version 3.0.1)
How do I specify the hosts on which my MPI job runs?
Related
While I was running kubectl command in my ubuntu 16.04 os which is a 32 bit machine, I was getting
cannot execute binary file: Exec format error
Can some one tell me whether Kubernetes works on 32 bit machine or not ?
Currently there are no ready-made binaries for 32bit systems at: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.7.md#downloads-for-v1710
You can build kubernetes from source though: https://kubernetes.io/docs/getting-started-guides/binary_release/#building-from-source
As a commenter mentioned, there is support for 32bit systems for the client tool, kubectl: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.7.md#client-binaries
Can anyone please guide me step by step installation one by one for Datastax Enterprise Installation on Virtual Box CentOS .
I checked Datastax Documentation , but getting little bit confused in few steps and due to which I am not satisfied. Also checked other resources but not able to understand completely.
So Help me to know installation process one by one with all basis steps.
Thanks in advance .
You may have an easier time using OpsCenter's Lifecycle Manager to deploy DSE. (Disclaimer, I am a Lifecycle Manager dev so am biased.)
First you need to install OpsCenter in a separate VM or Centos box. If you're able to get through the Java install and yum repository setup parts of DSE setup, this won't be difficult: https://docs.datastax.com/en/opscenter/6.0/opsc/install/opscInstallRHEL_t.html
Then run an install job from LCM: https://docs.datastax.com/en/opscenter/6.0/opsc/LCM/opscLCMinstallJob.html Example the pre-requisite section of that page carefully. It will show you the things you need to do in LCM to get ready to run the job, it's all point-and-click, though.
The only pre-requisites on your target DSE machine are "python" (usually installed by default) and for the minute "which", though we'll be removing that dependency in an upcoming version.
Note at the end of this process, you'll need to provide cqlsh an IP address, username, and password to connect to the cluster... even when making a "local" connection from your DSE vm. For example: "cqlsh 192.168.1.100 -u cassandra -p the-password-you-chose-during-lcm-install"
I have a spark cluster running on 10 machines (1 - 10) with the master at machine 1. All of these run on CentOS 6.4.
I am trying to connect a jupyterhub installation (which is running inside a ubuntu docker because of issues with installing on CentOS), using sparkR, to the cluster and get the spark context.
The code I am using is
Sys.setenv(SPARK_HOME="/usr/local/spark-1.4.1-bin-hadoop2.4")
library(SparkR)
sc <- sparkR.init(master="spark://<master-ip>:7077")
The output I get is
attaching package: ‘SparkR’
The following object is masked from ‘package:stats’:
filter
The following objects are masked from ‘package:base’:
intersect, sample, table
Launching java with spark-submit command spark-submit sparkr-shell/tmp/Rtmpzo6esw/backend_port29e74b83c7b3 Error in sparkR.init(master = "spark://10.10.5.51:7077"): JVM is not ready after 10 seconds
Error in sparkRSQL.init(sc): object 'sc' not found
I am using Spark 1.4.1. The spark cluster is also running CDH 5.
The jupyterhub installation can connect to the cluster via pyspark and I have python notebooks which use pyspark.
Can someone tell me what I am doing wrong?
I have a similar problem and have searching all around but no solutions. Can you please tell me what do you mean by "jupyterhub installation (which is running inside a ubuntu docker because of issues with installing on CentOS), "?
We have 4 clusters too on CentOS 6.4. One of my other problem is that how do use an IDE like IPython or RStudio to interact with these 4 servers? Do I use my laptop to connect to these servers remotely (if yes, then how?) and if no then what can be the other solution.
Now to answer your question, I can give it a try. I think the you have to use --yarn-cluster option as stated here I hope this helps you solving the problem.
Cheers,
Ashish
I am a newcomer to Hadoop. For my College project we are given 4 VMs. I need to configure a multi-mode Hadoop cluster on this ( 1 master 3 slaves) and run my webapp on it. I would be using HBase in my project. Usually CentOS is used for installation and deployment of HDP, whereas I was given ubuntu. I cannot use Apache ambari plugin for installation as it is not supported in Ubuntu. I need to manually deploy them, Hence I tried looking out for tutorials.
I looked out for a tutorial to install HDP multinode clusters on ubuntu and found this [http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/]
But its too outdated (2010)
I have the official documentation here, but I am not able to follow it properly.
[http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-latest/bk_installing_manually_book/content/rpm-chap2-3.html] and I tried following them.
Could someone suggest me somelinks which are latest, a tutorial with decent amount of screenshots for installation of multinode clusters over Ubuntu 14.04 ( 12.04 is also fine).
Thanks a lot!!
The Michael Noll tutorial is too old, I think. I found this site:
https://www.digitalocean.com/community/tutorials/how-to-install-hadoop-on-ubuntu-13-10
I have a mini cluster (with 5 slaves and a master) in my University Lab. Ubuntu 12.04 and Hadoop 2.5.0 is there. Furthermore, I have a VM cluster in my laptop (2 slaves and a master) of Hadoop 1.2.1 on Ubuntu 12.04 too.
But I couldn't install Hadoop (any version) in Ubuntu 14.04. I don't remember the cause, but I think it was some problem with Java version (I don't check that).
I hope that help you!
I can across the same issue to install HDP 2.2 on Ubuntu 14.04, and found a solution.
I documented everything here: http://www.swiss-scalability.com/2014/12/install-hdp-22-on-ubuntu-1404-trusty.html
In a nutshell, the magic happens here:
sed -e "s/14.04/12.04/g" -i /etc/*-release
And the you can install or restart ambari-agent, it will be able to communicate with ambari-server.
I am trying to follow this tutorial for building and running an MPI application on an ARM based Ubuntu 11.10 system.
When installing open-mpi environment on my PC machine, the sample program runs well. However, trying the same on the ARM machine, the terminal hangs up and I need to kill the MPI process from a second terminal in order to release it.
The MPI packages I installed using apt-get, on both machines, were mpi-default-dev and mpi-default-bin, so I assume that the packages are as updated as they can be.
The first sample program in the tutorial makes every process prints a "hello" message with some info. On the PC I get messages from all 8 processes (although running on a single core) and then the program ends. On the ARM, I get no output at all. The program is just stuck immediately after launch.
Any idea on what's wrong? I am not sure even where to start to debug this?
Update: I tried removing the OpenMPI package and install the alternative MPICH2 package - but the result is just the same.
Ubuntu 11.10 did not ship with a functional Open MPI implementation for ARM (although it may have shipped with a nonfunctional one). Ubuntu 12.04 did.
I would recommend building your own Open MPI from source - available at http://www.open-mpi.org/software/ompi/v1.6/, unless you can update to a more recent version of Ubuntu.
Alternatively, you could rebuild the 11.10 package using the fixes pointed out in https://bugs.launchpad.net/ubuntu/+source/openmpi/+bug/949044.