I would like to set up an apache spark Standalone cluster using two different computers (windows 10) of the same wifi network as slaves.
I can successfully set up the master using Windows bash(linux subsystem) and connect a slave from the same machine.
The problem is when I'm trying to connect a slave from a different machine (same wifi) using the local ip of the master. I even temporarily opened up all the ports of both the machines and mamaged to view the spark ui from the slave machine using the local ip of the master. But still trying to set up a slave on master-local-ip:7077 didn't work.
I believe that the correct way is to use password-less ssh between them but I'm not sure how to configure it for local ip addresses using the windows bash. I tried to create the key and when I executed ssh-copy-id the connection refused.
Any suggestions?
Related
I have Jmeter in local windows & application is running in VM Linux.
Note: JMeter is not available in VM machine.
I need to establish connection to VM machine & thereby publish the data in MQTT broker in VM machine.
for VM connection, I have done below changes:
jmeter.properties: I have included:
remote_hosts=XX.XX.XXX.XX:XX
user.properties, i have included:
server.rmi.ssl.disable=true
And when i perform remote run, i could see the error:
"non-jrmp server at remote endpoint"
Am not sure, do I miss any points.
You don't understand the concept of Distributed testing, defining remote_hosts property is needed when you need to connect JMeter to another JMeter so 2 (or more) JMeter instances would act as a single load generator, people normally do it when single JMeter cannot generate the required load because machine is not powerful enough.
If you have JMeter installed on your Windows machine and MQTT broker is running on Linux machine just install MQTT Protocol Support plugin using JMeter Plugins Manager:
once done you will have MQTT Connect sampler where you will be able to configure your Linux machine IP address and port in order to establish connection with it:
More information: Testing the MQTT Messaging Broker for IoT - A Guide
I have 2 PC (with Windows as default OS), let's say PC_master and PC_slave. Since it is recommended to work with Linux to manage a spark Cluster i created one Virtual Machines with Linux on each of theses PC: VM_master on PC_master and VM_slave on PC_slave.
I would like to setup a Standalone Spark Cluster, so which Networking configuration should i use (NAT, Bridge, HostOnly?) for theses 2VM in order that they own their private IP adress letting me ssh from one to another without the need of port forwarding ?
Edit: My two laptop are linked to the same box, so they have private IP and supposed to be linked to the same router.
Finally it seems that if i want to give each VM its own Ip adress to create a private Network i have to choose Bridge Network which will assign a unique Ip for each VM!
I'm doing this for the very first time and I need help about clearing some stuffs.
I have Ubuntu 14.04 desktop on one machine, and on the other I have Windows 8.1. also on both machines I have installed the same version of Cassandra.
Can someone tell me, is it possible to connect 2 desktop machines and make a cluster with 2 nodes in Cassandra and how?
You need make sure the following settings are set correctly in the cassandra.yaml on each machine:
cluster_name - this needs to be the same on both nodes
seed_provider.parameters.seeds - this needs to be set to the external IP address of one of the nodes and needs to be the same on both nodes.
listen_address - this needs to be set to the external IP address on each machine.
Note: Make sure you can ping each machine from the other on the IP address you use for the listen_address and make sure that the storage_port 7000 isn't blocked by a firewall on either machine.
I have asked another question here, but have since narrowed down my issue to be using the "host-only" network adapter, versus a bridge network (with DHCP).
I have created two Ubuntu virtuals, and have installed Cassandra (from DataStax) the same way.
The first box is using a host-only adapter...
dev.vm.network :private_network, ip: "192.168.10.200"
And the second is bridged using DHCP...
dev.vm.network :public_network
The box using a "host-only" private network doesn't allow me remote access, even though the Cassandra configuration is the same. The box using a public network has remote access working just fine.
I am using Windows 10, and I am not sure if this was working on Windows 8.
I have a GPU cluster which I can access by ssh-ing into an intermediate computer, and then ssh-ing into the cluster. (The intermediate computer resides on the same network as the cluster, but the cluster is not connected to the Internet)
I prefer not to use in-terminal editors like vim and would also like to avoid copying files to the intermediate computer and then onto the cluster, so I am looking for a way to remotely edit files on the cluster with an IDE on my local computer.
If I were directly ssh-ed into the cluster, I could use SFTP to edit the files on the cluster.
However, there is an intermediate computer. Is there someway I can use SFTP or a similar technology to achieve my goal?
Thanks for any help and let me know if I can be more clear.
You are looking for an SFTP connection to your cluster through a SSH connection to your gateway - so for SSH tunneling.
Use on your local machine
ssh gateway -L localPort:cluster:portToCluster -l user -p portToGateway
to open a SSH tunnel to your cluster. For localPort use any free port on your local machine (e.g. 2222), portToCluster is the port you usually use for connecting to your cluster from the gateway (standard port is 22) and portToGateway is the port you use for connecting to the gateway from your local machine.
Now you can access the cluster simply using a SFTP connection to localhost:2222
Your SSH tunnel must be open as long as you wish to use this SFTP connection. So keep your SSH connection running.