I've been studying Linux socket programming recently, and the concepts are still swirling and unsettled in my head. Can someone confirm or correct my understanding of the domain argument to socket() and socketpair(): one should choose PF_LOCAL (or PF_UNIX) if one wants the socket communication to be strictly within the same computer, and one should choose PF_INET if the socket communication is meant to be between different computers -- is that correct?
No, it's the communications domain you want to use. See the man page for socket. For example, AF_INET means v4 internet protocols, AF_INET6 means v6 internet protocols, AF_APPLETALK means AppleTalk, and so forth. You almost certainly want AF_INET or AF_INET6.
Whether the other program you'll be communicating with is on the same machine or not isn't really relevant since you can communicate with the local host just fine using internet protocols.
However, there is a small performance penalty associated with using the internet domain protocols. If your application will be connecting only with other applications on the same machine, using the AF_LOCAL/AF_UNIX domain will be faster and will offer you some additional advantages such as file-level security controls on the sockets. Just be aware that you won't be able to use your code between different computers without modifying it if you go that route.
A good discussion of the pros and cons of this choice can be found here.
Related
I'm breaking a big application into several processes and I want each process to communicate with each other.
for now it's gonna be on the same server, but later several servers on same local network will have several processes that will need to communicate between each other. (means service on one server, with service on other server on same vpc)
so.. my raw options are tcp or unix sockets. I know that with Unix sockets can be useful only if you're on the same server. but we're thinking about writing our own implementation that on same server processes will communicate on unix sockets, and between servers that will communicate using tcp.
is it worth it ? of course tcp sockets are slower then unix sockets.. cause it doesn't go through the network and doesn't get wrapped with tcp related data. the question is by how much ? I couldn't find online proof of benchmarking between tcp and unix sockets. if tcp adds 3%-5% overhead that's cool, but can it be more then that ? I'd like to learn from experience of big projects.. of other people over the years, but didn't find anything relevant.
next...
our project is a NodejS project.
some people may say that I can use a broker for messages, so I tried using nats.io compared to node-ipc (https://www.npmjs.com/package/node-ipc) and I found out that node-ipc is 4 times faster but nats has the cool publish-subscribe feature... but performance is important.
so I have tons of options, no concrete decision.
any information regarding the issue would be greatly appreciated.
The question is actually too broad to answer, but one answer for TCP vs unix domain sockets:
Architect your code, so that you can easily move between those if necessary. The programming model for these is basically the same (both are bidirectional streams of data), and the read/write APIs on OS level as well as in most frameworks is the same. This means e.g. in node both will inherit from the Readable/WriteableStream interfaces. That means the only code that you need to change for switching between those is the listener on the server side where you call the TCP accept APIs instead of the unix domain socket accept APIs and the other way around. You can even have your application accept both types of connections and later on handle them the same internally.
TCP support is always nice because it gives you some flexibility. With my last measurement the overhead was a little bit more (I think 30% versus TCP over loopback) but these are all micro benchmarks and it won't matter for most applications. Unix domain sockets might have an advantage if require some of their special functions, e.g. the ability to send file descriptors across them.
And regarding TCP vs NATS & Co:
If you are not that experienced with network programming and protocol design it makes sense to use readymade IPC systems. That could be anything from HTTP to gRPC to Thrift. These are all point-to-point systems. NATS is different, since its a message broker and not RPC. It also requires an extra component in the middle. Whether this makes sense totally depends on the application.
I need a way for my application to find someone with the same application on a local network.
But can't find the docs to use a protocol like SSPD. It says its UDP on port 1900 and the ip 239.255.255.250.
It also says it can use NOTIFY and M-SEARCH.
But that is it. I've hit a wall. And I need some guidence.
OVERVIEW
Trying to make my apps automatically find each other and exchange ip/names on a local network so they can talk to each other p2p-style without a connection broker.
The UPnP Device Architecture reference contains the de-facto SSDP reference: it's very similar to the IETF spec and has dozens of well tested implementations already. Chapter 1 contains everything needed to implement SSDP, assuming you know how to send and receive multicast and unicast UDP messages.
My advice on implementing SSDP: Don't. It may look simple but there are a lot of gotchas on the way. Find a library or service that does SSDP for you. If you are on linux I suggest GSSDP (reference, code) (but I'm a bit biased having worked on it). I would assume other platforms have similar libraries.
I am building a message layer for processes running on an embedded Linux system. I am planing to use sockets. This system might be ported to different operating systems down the road so portability is a concern. Performance is below portability in priority order.
I have a few questions regarding my way forward.
I am thinking of using internet sockets over TCP/IP for this communication between local processes for the sake of portability. Is there any reason that I should not do that and use domain sockets?
Does it really improve the portability when using internet sockets instead of domain sockets?
If this is indeed the way forward, can you point me in the right direction (how to use ports for each process etc.) with some online resources?
Need a basic direction in the following project.
There is a linux based controller doing some industrial control stuff.
The box is equipped with cellular modem and is capable to get online through cellular carrier. Cellular communication is used because controller is mostly installed where no cables or short range radio is available. Places where sun don't normally shine :)
The task is to allow internet clients to connect directly to the box for some basic control/monitoring stuff. The problem is connectivity - how clients will discover the box? - I'd like to have the box act as a server (if possible). Assuming that cellular carrier allows the box to get online doesn't necessarily mean that the box will get public IP so that anyone would be able to get connected. To my understanding the cellular network acts as a gateway from those who are working inside of it, and reaching someone in that network from outside isn't possible. Am I wrong? We are looking for a generic solution, not a solution around particular cellular provider. The controller is installed in different countries, we need to find the standard way to "webify" it.
The software (and hardware) in the box is ours, we can basically do anything, but I am looking for the right way to do it in order to avoid surprises with different providers later. BTW, the solution doesn't necessarily have to be technical, may be it's possible to buy a permanent IP's per box, or setup VPNs.. Which way should I dig to? What questions to ask?
Your ideas are welcome!
Your summary of the problem is basically correct. I've implemented several systems that do this, and the odds of success are good.
The way you tackle this will depend on the number of remote units you expect a single user to interact with. If each user will handle only one or two devices, it's plausible to implement the web server on the remote device. If each user handles many devices, consider centralising as much administration as possible. I've implemented this using Zenoss for data logging, and a custom control server.
If the web server sits on the remote device, you can either buy a SIM with a static IP, or use a proxy server. I recommend setting up a proxy server unless the number of devices is very small.
There are three options for SIMs:
Static IP with an address on the public Internet will be expensive, and negotiating the deal with each provider in each country will be irksome. No proxy server is required.
Private APN SIMs will give you the option of a static address, but in a private address range. Negotiation with the mobile network is still required, and you will require a proxy server to sit between the public Internet and the private address range,
Standard data SIMs will connect to the Internet through NAT. You can use these to host your service by opening a VPN connection (we used openvpn) to your server. You can now reach the devices directly by connecting to the same VPN, or through a proxy server.
If you use openvpn, here are some more tips:
Give each unit a public serial number, and a private key. Store these in the firmware of the unit, and in a central database. Put the public serial number on the outside of the unit. You can use an openvpn login script to ensure that a particular unit always appears at the correct IP address, which keeps the proxy configuration static.
You can control openvpn's bandwidth usage by adjusting its keepalive behaviour, and how often it renegotiates. Measure and tune this before a large deployment.
The NAT timeouts in the mobile networks are generally between 5 and 15 minutes. The device must send a packet to the server often enough to keep NAT alive.
Cheap SIM deals may be web only with limited ports.
Other tips:
GPRS modem firmware can (rarely) crash internally. If your hardware supports it, provide software with the ability to power cycle the modem.
Test your box in areas with poor coverage in your own country before you send out international shipments.
This is a typical problem with "mobile agent" appearing in different places or using different providers (in this case just one provider, but it's almost the same). Usually it's solved using some kind of home agent - a server that the mobile connects to and gives details about how to reach it or if it can't be reached directly then the home agent acts as a proxy.
Client always contact the home agent first and then if it is possible they contact the mobile or if it's not they use the server as a proxy.
In some cases dynamic dns might be sufficient in other you need real proxy/ façade.
There's a good book: Andrew S. Tanenbaum & Maarten van Steen :"Distributed Systems: Principles and Paradigms"
You can ask cellular provider to give you a SIM card with internet access and fixed IP address. Then you can host any server you like. Do not forget that you are dealing with limited bandwidth.
We have an embedded device that needs to interact with an enterprise software system.
The enterprise system currently uses many different mechanisms for communication between its components: ODBC, RPC, proprietary protocol over TCP/IP, and is moving to .Net-implmented web services.
The embedded device runs a flavor of *nix, so we're looking at what the best interaction mechanism is.
The requirements for the communication are:
Must run over TCP/IP.
Must also run over RS-232 or USB.
Must be secure (e.g. HTTPS or SSL).
Must be capable of transferring ~32MB of data.
Our current best option is gSOAP.
Does anyone out there in SO-land have any other suggestions?
Edit: Steven's answer gave me the most new pointers. Thanks to all!
You can define RESTful services the use HTTPS (which uses TCP/IP by definition) and is capable of transferring any amount of data.
The advantage of REST over SOAP is that REST is simpler. It can use JSON instead of XML which is simpler.
It has less overhead than the SOAP protocol.
Can't you just use SSL over TCP?
If you have some kind of *nix (may I guess? It's either QNX or embedded linux, right?) it should work pretty much out of the box via Ethernet, USB and RS232. Keep thing simple.
32mb is plenty of memory for this task. I would allocate between 2 and 4 mb of memory for networking & encryption (code + data).
It's not real clear why you want to tie this to a remote-procedure-call protocol like SOAP. Are there other requirements you aren't mentioning?
In general, though, this sort of thing is handled very easily using normal web-based services. You can get very lightweight http processors written in C; see this Wikipedia article for comparisons of a number of them. Then a REST interface will work fine. There are network interfaces that treat USB as a TCP connection, as well.
If you must be able to run over RS232, you might want to look elsewhere; in that case, something like sftp might do better. Or write a simple application-layer protocol that you can run over an encrypted connection.
If you are going to connect your application using RS232, I assume that you will be using PPP to connect the device to the internet. The amount of data that you are proposing to transfer is somewhat worrisome, however. Most RS232 connections are limited to 115200 baud which, ignoring the overhead required for TCP/IP/PPP framing is going to yield a transfer rate of at most 11,000 bytes per second. This implies that it will take a minimum of approximately 2800 seconds or 46 minutes to make whatever transfer that you intend.