I'm breaking a big application into several processes and I want each process to communicate with each other.
for now it's gonna be on the same server, but later several servers on same local network will have several processes that will need to communicate between each other. (means service on one server, with service on other server on same vpc)
so.. my raw options are tcp or unix sockets. I know that with Unix sockets can be useful only if you're on the same server. but we're thinking about writing our own implementation that on same server processes will communicate on unix sockets, and between servers that will communicate using tcp.
is it worth it ? of course tcp sockets are slower then unix sockets.. cause it doesn't go through the network and doesn't get wrapped with tcp related data. the question is by how much ? I couldn't find online proof of benchmarking between tcp and unix sockets. if tcp adds 3%-5% overhead that's cool, but can it be more then that ? I'd like to learn from experience of big projects.. of other people over the years, but didn't find anything relevant.
next...
our project is a NodejS project.
some people may say that I can use a broker for messages, so I tried using nats.io compared to node-ipc (https://www.npmjs.com/package/node-ipc) and I found out that node-ipc is 4 times faster but nats has the cool publish-subscribe feature... but performance is important.
so I have tons of options, no concrete decision.
any information regarding the issue would be greatly appreciated.
The question is actually too broad to answer, but one answer for TCP vs unix domain sockets:
Architect your code, so that you can easily move between those if necessary. The programming model for these is basically the same (both are bidirectional streams of data), and the read/write APIs on OS level as well as in most frameworks is the same. This means e.g. in node both will inherit from the Readable/WriteableStream interfaces. That means the only code that you need to change for switching between those is the listener on the server side where you call the TCP accept APIs instead of the unix domain socket accept APIs and the other way around. You can even have your application accept both types of connections and later on handle them the same internally.
TCP support is always nice because it gives you some flexibility. With my last measurement the overhead was a little bit more (I think 30% versus TCP over loopback) but these are all micro benchmarks and it won't matter for most applications. Unix domain sockets might have an advantage if require some of their special functions, e.g. the ability to send file descriptors across them.
And regarding TCP vs NATS & Co:
If you are not that experienced with network programming and protocol design it makes sense to use readymade IPC systems. That could be anything from HTTP to gRPC to Thrift. These are all point-to-point systems. NATS is different, since its a message broker and not RPC. It also requires an extra component in the middle. Whether this makes sense totally depends on the application.
Related
I've been studying Linux socket programming recently, and the concepts are still swirling and unsettled in my head. Can someone confirm or correct my understanding of the domain argument to socket() and socketpair(): one should choose PF_LOCAL (or PF_UNIX) if one wants the socket communication to be strictly within the same computer, and one should choose PF_INET if the socket communication is meant to be between different computers -- is that correct?
No, it's the communications domain you want to use. See the man page for socket. For example, AF_INET means v4 internet protocols, AF_INET6 means v6 internet protocols, AF_APPLETALK means AppleTalk, and so forth. You almost certainly want AF_INET or AF_INET6.
Whether the other program you'll be communicating with is on the same machine or not isn't really relevant since you can communicate with the local host just fine using internet protocols.
However, there is a small performance penalty associated with using the internet domain protocols. If your application will be connecting only with other applications on the same machine, using the AF_LOCAL/AF_UNIX domain will be faster and will offer you some additional advantages such as file-level security controls on the sockets. Just be aware that you won't be able to use your code between different computers without modifying it if you go that route.
A good discussion of the pros and cons of this choice can be found here.
I need a way for my application to find someone with the same application on a local network.
But can't find the docs to use a protocol like SSPD. It says its UDP on port 1900 and the ip 239.255.255.250.
It also says it can use NOTIFY and M-SEARCH.
But that is it. I've hit a wall. And I need some guidence.
OVERVIEW
Trying to make my apps automatically find each other and exchange ip/names on a local network so they can talk to each other p2p-style without a connection broker.
The UPnP Device Architecture reference contains the de-facto SSDP reference: it's very similar to the IETF spec and has dozens of well tested implementations already. Chapter 1 contains everything needed to implement SSDP, assuming you know how to send and receive multicast and unicast UDP messages.
My advice on implementing SSDP: Don't. It may look simple but there are a lot of gotchas on the way. Find a library or service that does SSDP for you. If you are on linux I suggest GSSDP (reference, code) (but I'm a bit biased having worked on it). I would assume other platforms have similar libraries.
I am building a message layer for processes running on an embedded Linux system. I am planing to use sockets. This system might be ported to different operating systems down the road so portability is a concern. Performance is below portability in priority order.
I have a few questions regarding my way forward.
I am thinking of using internet sockets over TCP/IP for this communication between local processes for the sake of portability. Is there any reason that I should not do that and use domain sockets?
Does it really improve the portability when using internet sockets instead of domain sockets?
If this is indeed the way forward, can you point me in the right direction (how to use ports for each process etc.) with some online resources?
Are there any tools to dynamically and graphically display the iterations between a bound socket and connected clients? Debugging issues in sockets with multiple simultaneous interactions can be a huge mess, I would think that displaying the interactions graphically would be a great help to understanding what is going on in a server's interactions.
I'm unaware of graphical applications per se.
However, tcpdump or (for those who require a GUI) wireshark are pretty good at showing you the packets being sent, which is what you actually want here in general.
If looking at the big picture is enough for you - process hierarchy and the connections (TCP socket, Unix domain socket, pipes) between them - you can give a try to ipcvis:
https://sourceforge.net/projects/ipcvis/
The tool records process hierarchy and socket information for distinct states and then visualize new relations from state to state:
http://youtu.be/8XFKwzkexQY
We have an embedded device that needs to interact with an enterprise software system.
The enterprise system currently uses many different mechanisms for communication between its components: ODBC, RPC, proprietary protocol over TCP/IP, and is moving to .Net-implmented web services.
The embedded device runs a flavor of *nix, so we're looking at what the best interaction mechanism is.
The requirements for the communication are:
Must run over TCP/IP.
Must also run over RS-232 or USB.
Must be secure (e.g. HTTPS or SSL).
Must be capable of transferring ~32MB of data.
Our current best option is gSOAP.
Does anyone out there in SO-land have any other suggestions?
Edit: Steven's answer gave me the most new pointers. Thanks to all!
You can define RESTful services the use HTTPS (which uses TCP/IP by definition) and is capable of transferring any amount of data.
The advantage of REST over SOAP is that REST is simpler. It can use JSON instead of XML which is simpler.
It has less overhead than the SOAP protocol.
Can't you just use SSL over TCP?
If you have some kind of *nix (may I guess? It's either QNX or embedded linux, right?) it should work pretty much out of the box via Ethernet, USB and RS232. Keep thing simple.
32mb is plenty of memory for this task. I would allocate between 2 and 4 mb of memory for networking & encryption (code + data).
It's not real clear why you want to tie this to a remote-procedure-call protocol like SOAP. Are there other requirements you aren't mentioning?
In general, though, this sort of thing is handled very easily using normal web-based services. You can get very lightweight http processors written in C; see this Wikipedia article for comparisons of a number of them. Then a REST interface will work fine. There are network interfaces that treat USB as a TCP connection, as well.
If you must be able to run over RS232, you might want to look elsewhere; in that case, something like sftp might do better. Or write a simple application-layer protocol that you can run over an encrypted connection.
If you are going to connect your application using RS232, I assume that you will be using PPP to connect the device to the internet. The amount of data that you are proposing to transfer is somewhat worrisome, however. Most RS232 connections are limited to 115200 baud which, ignoring the overhead required for TCP/IP/PPP framing is going to yield a transfer rate of at most 11,000 bytes per second. This implies that it will take a minimum of approximately 2800 seconds or 46 minutes to make whatever transfer that you intend.