Problem: I am developing a graphical front end for a distributed CPU/GPU simulator. As this simulator utilizes MPI, it requires a hostfile detailing the hostnames for all computers being used on the network so that it knows what machines to distribute across. As the end users for my application are not computer scientists (and may not even be very computer literate), I can't expect them to know/find the hostnames of every computer on their network/cluster. I would like to programmatically perform this hostname discovery so that, upon application start-up, the user can see the available machines, and from those, pick the hosts they want to run on. If possible, I would like this solution to be cross platform but as the simulator currently contains some linux dependencies I can deal with a Linux only solution.
What I have tried so far: I tried utilizing the nmap package to discover hosts on a network with commands like nmap -sP <ip address range> using the ip address range for that local network. However, it only dumps the IP addresses for the hosts (not the host names) and I'm not sure how to translate these IP addresses into ssh hostnames (as MPI uses ssh for host discovery). Additionally, I used a similar approach with ping supplying the broadcast address and it returned nearly identical results.
I apologize for the broad nature of this question and the lack of code shown but I am not very experienced with network probing / programming and I am really not even sure where to start. I tried googling this but I was unable to find a suitable option (possibly because my lack of experience caused me to use improper terminology triggering improper results) My background is primarily in graphics and user interface programming, so this is a little beyond my comfort zone.
SSH doesn't care if it is given hostnames or IP addresses to connect to (not sure if this applies when there are host-specific configurations). Most MPI implementations don't care too, e.g. in Open MPI connection URIs addresses are all numeric, so a hostfile with IPs would be fine. HTTP servers on the other hand care because of the virtual hosting thing where many different sites resolve to the same IP address but the server is supplied the actual hostname via the Host HTTP header.
Unsolicited advice: finding hosts by ping is fine, but it doesn't guarantee that you have found machines, where SSH is running. You would better scan for systems with port 22 open that accept TCP connections:
$ nmap -oX -sT -p22 <ip range>
-oX produces XML output that can be easily parsed. -oG is also a nice format for automated parsing of the scan results. Also having SSH running doesn't necessarily mean that the user would be able to log into the system - for example it could be a network router or another remotely manageable device. One also has to take care of only showing machines where the user can log on without having to supply a password, e.g. with RSA/DSA public keys, otherwise starting an MPI job would be a really tedious task. You can test each host found with something like:
$ ssh -2 -o "PreferredAuthentications=gssapi-with-mic,hostbased,publickey" \
<host> hostname
This command basically excludes all interactive authentication methods. If connection succeeds, it will output the hostname of the remote machine. Otherwise you'd get a permission denied error and a non-zero exit code from the SSH client.
Related
I want to connect to a locked (cannot be modified and I do not have permissions to log into) windows vm which is hosted in a linux machine. Until now these two machines were communicating via port 2277. However for security reasons, the port 2277 is only accessible via localhost (127.0.0.1).
The original proposed solution was to use ssh-tunneling. However since the hosted windows vm will always stay with the linux machine, so I was thinking something simpler.
This Windows virtual machine has ip 192.168.0.1 and the default gateway is 192.168.0.2. The later is the ip address that my linux machine can see.
After searching the internet I tried
socat TCP4:192.168.0.1:2277,reuseaddr,fork TCP4:127.0.0.1:2277
as well as some other random combinations without success.
My understanding is that this failed because for socat to work both sockets must be open.
However the first one is not open by default (checked with ss -ltn) as I need to run the windows service first (which it cannot run as it cannot communicate with iphost:2277)
Any ideas on how to proceed?
Socat provides the retry=N and forever options to handle situations like this. Thus, try something like this:
socat -d -d TCP4:192.168.0.1:2277,reuseaddr,fork,forever TCP4:127.0.0.1:2277
With the interval=<seconds> option you can specify how long Socat waits after each failed attempt.
It turns out that the command that I wanted was the following
socat tcp-listen:2277,bind=192.168.0.1,fork,reuseaddr tcp:127.0.0.1:2277
The retry=N and forever options could be also useful.
First of all, I wanna say that I don't have much experience in advanced networking on Linux.
A have a task to deploy our .deb packages in containers, and applications are mostly tunned for operating on localhost while being designed with the capability of operating onset of server machines (DB, application, client, etc), but since components of the app have been distributed between containers, I need to make it work together. The goal is to do it w/o any pre-setup sequences that change the IP address in configs for components since target IP is uncertain and IP alias in /etc/hosts may not solve the problem.
Could I somehow intercept outbound connection to localhost:5672 and forward it to, we say, 172.18.0.4:5672 with the ability to correctly receive incoming traffic from the resource we forwarded to? Can you give me some examples of the script?
My setup is that I have a machine running little eclipse server in node, and I want to configure other machines on the LAN to run a website off of that first computer. Everything is working fine, but I have to manually tell the other computers which local IP address to use when opening chrome.
All computers at this point are running ubuntu.
Ideally I'd like to make a bash script on any new computer which finds the IP address of the computer running the express server, then opens chrome at that address.
Googling tends to get me answers for the question 'how to see all computers on my LAN'. It seems that I can achieve that in many ways, most notably nmap.
NMap works fine, but now my question is how I can make the server computer broadcast its status as the desired machine, then extract its sepcific IP address?
Some of the other possibilities I've considered are the following (I state why I'm not sure they're right for me, but please correct me if I'm wrong):
Hosts File
This requires setup for each collection of computers, which is undesirable
DHCP+DNS on router
Again requires set up on the router, which can't be guaranteed to have the required functionality every time.
DNS server on a machine
Again this requires knowing the IP address of the server
What I'd really like is for the process to be automatic – the server machine is happily running its local website, then any other computer that joins the network is able to find it then open that website in chrome, without the person installing the computer having to know the server's IP address. Is such a thing feasible?
Scenario:
Many embedded devices (running Linux) out in the fields, behind routers so NAT'd and we can't make connections to them.
We need for a support person to be able to initiate a terminal/shell session on any of the devices.
Their local terminal will also be NAT'd behind a corporate firewall, so we need some central "meeting point" that both they and the device can connect to.
If necessary, we could require the support person to log into some sort of terminal server, but I'd prefer a solution that just popped up a terminal window on their desktop.
We can (through other means) tell the device to execute some arbitary script or application to start up the session.
Without the NAT, it's just SSH to the device and away we go. But what are my options in this NAT'd environment?
We're OK to develop code at either end or at the meeting point server if required, but obviously if there are apps out there so we don't have to write stuff, even better.
Pointers to other questions I may have missed (although I have looked) or to applications that I should consider for the central "meeting point" server welcomed
How about simply setting up an ssh server that is reachable by both the device and the support user, and have the device set up a reverse tunnel (using remote port forwarding)?
ssh -R 10022:localhost:22 device#server
Then the support personnel can simply connect to the server and log on using
ssh -p 10022 localhost
Of course there are several security aspects that need to be accounted for here, depending on what kind of information the devices hold/have access to and how the support organization is set up.
SSH is an adequate tool for this. You will, as you say, need a middle-man server. But it would be very easy to set up, assuming that your 'other means of executing a script' are remote and can be executed from your office.
So, fire up a new server on a global IP (an Amazon AWS micro node is free for a year and would do the job just fine), and install an ssh deamon. Say it has the hostname middleman.example.org.
The script to put onto your embedded devices would look like;
#!/bin/bash
ssh -i ./middle_id.pem -R 22:localhost:2222 middleuser#middle.example.org
(The private key authentication would be a way of making the login non-interactive)
The script to put onto your desktop machines would look like; (assuming the argument $1 is the IP of the embedded device, and that prod_remote_device.sh executes the above script on the chosen embedded device.)
#!/bin/bash
./prod_remote_device.sh $1
ssh -i ./device_id.pem deviceuser#middle.example.org:2222
And that should forward your connection to the embedded device.
In order to make it bind to all interfaces, use:
ssh -N -R 0.0.0.0:2222:localhost:22 root#example.com
Don't forget to edit /etc/ssh/sshd_config and go to GatewayPorts and enable it and set it to yes.
And Then connect to it from any Loopback or Ethernet interface.
I need to store persistent reference to third party device on an arbitrary IP network where the IP address of the devices may be static or randomly assigned by DHCP. I don't control the devices on the network and I can't rely on DNS and other ad-hoc networking protocols existing or working with the devices.
So I have been instructed to investigate using hardware addresses and ARP. This will work but I don't want to duplicate code. The kernel must manage an ARP table. On Windows you can access it using GetIpNetTable etc.
I am hoping there is an API to answer these two questions:
How do I translate from IP to MAC address? (ARP)
How do I translate from MAC to IP address? (InARP)
If not then I may have to do it more manually:
How do I read the kernel's ARP table?
How do I add an entry if I have the determined a mapping myself?
/proc/net/arp
K
ARP tables tend to be fairly local and short-lived. If you examine the protocol, the real MAC addresses are generally only provided when the given IP address is in the local subnet.
Otherwise, the packet is forwarded to the local router, which is then responsible for forwarding it.
If you do "arp -g" on Windows or "arp -a" on UNIX, you'll see the table, but I don't think it will do you any good, due to the reasons mentioned above. That command and
That's really what DNS is for but, as you say, it may not be an option for you.
You may well have to write your own 'ARP' database at your application level.
As for ARP:
You could use system("/usr/bin/arp -option_of_choice"); and parse the output, but that's an ugly hack. -- Not my recommendation.
Take a look at /usr/include/linux/sockios.h -- At the SIOCGARP, SIOCDARP, and SIOCSARP details. Those are ioctls that you can perform to manage the ARP table on linux. Of course, you'll have to perform these ioctls on a socket fd.
Here's some examples: SIOCGARP examples
I'm sure you can find many other examples in several other languages as well. As I'm assuming that you're using C.
As for RARP:
A quote from the linux rarp manpage:
" This program is obsolete. From version 2.3, the Linux kernel no longer
contains RARP support. For a replacement RARP daemon, see ftp://ftp.demen-
tia.org/pub/net-tools"
So you'll have to install rarpd on the target system.