Everywhere I read about named pipes vs sockets, I see that a socket can do what a named pipe can, and more. I am guessing that pipes may be faster because they are simpler, but I haven't find it explicitly anywhere.
In Linux, is there any advantage in using named pipes instead of sockets?
Thanks for your help and time.
EDIT: I found some comparisons here, but these are UNIX-sockets (a.k.a. UNIX domain sockets), which based on this, I understand they are a different creature.
Therefore, I clarify: I am asking about TCP sockets versus Unix named pipes (because MS Windows has another thing called "named pipes", but they seem to be functionally more similar to UNIX domain sockets).
At this point, it looks more likely to me that UNIX named pipes are faster than TCP sockets, but I would like to know some technicalities on why that would the case, and other potential benefits.
This is likely an imperfect answer, but here's what I know.
Sending data over a TCP socket means that the transmission needs to through your networking system, will get a source & destination IP, split up in a maximum of 64K packets. Packaged in both a TCP and IP envelope, potentially go through firewall rules. The receiver also needs to acknowledge packages, make sure they arrive in order and there needs to be a negotation in place in case packages get lost and need to be re-sent.
Sending data in a named pipe more or less works like just writing to a file descriptor like STDOUT, and reading from STDIN.
Going through the network stack (even if it's just localhost) simply has a lot more layers and complexity. The system that can send a message reliably to the other side of the world needs that, but a simple local named pipe does not.
Everywhere I read about named pipes vs sockets, I see that a socket can do what a named pipe can, and more.
Named pipes are used like files.
This is a large benefit if you want to do something that works with a file but it does not work with a socket.
Examples:
ls > /dev/lp1
ls > ./myNamedPipe
# Not possible: ls > 127.0.0.1:9323
dd if=myFile bs=1 count=100 of=/dev/lp1
dd if=myFile bs=1 count=100 of=./myNamedPipe
# Not possible: dd if=myFile bs=1 count=100 of=127.0.0.1:9323
MS Windows has another thing called "named pipes", but they seem to be functionally more similar to UNIX domain sockets
The truth is that MS Windows has additional API functions for accessing named pipes. However, the standard Windows file API (that is used by C functions like open(), read() and write()) also works with named pipes.
For this reason you can also use a named pipe as file in Windows. I already did that to emulate a serial port with a certain device connected.
... and other potential benefits.
One obvious benefit (of both named pipes and Unix sockets) is the naming itself:
If some program foobarSimpleProgram wants to communicate to another program named foobarOtherProgram, it can simply create a Unix socket or a named pipe named /tmp/foobarProgramSuite.
It's very unlikely that any other program uses this name.
In the case of a TCP socket listening on localhost, the program must either use a fixed TCP port; in this case there is the risk that another program uses the same TCP port so only one of the two programs can be used.
Or the program binds to TCP port 0 so the OS assigns some free TCP port. Then the program writes the port number to a file /tmp/foobarProgramSuite which is read by the other program, which is then doing the connect().
This is more complicated than directly connecting to the pipe or socket with the given name.
Related
When writing from a laptop to an external serial device that I am using, it is possible for the laptop to be writing faster than the program running on the device can actually clear its serial read buffer, leading to data being lost do to the buffer overflowing.
Is there a way to simulate this behavior using a virtual serial port with socat? For my testing so far, I have used the following command:
socat -d -d pty,raw,echo=0 pty,raw,echo=0
(For the sake of the example, lets say it creates /dev/pts/2 and /dev/pts/3, referred to as A and B respectively).
Which creates two virtual serial ports that I can send data across. However, if I start writing a large amount of data to A without reading any from B, the writing to A will eventually block until I read from B and make room in its read buffer.
It is unclear to me whether there is some internal feedback between the read buffer of B, or if the actual blocking is occurring in the write buffer of A due to some other internal feedback within whatever socat setup that tells blocks the writing because there is no room in the read buffer of B.
Is there any way to avoid this blocking and have it overflow the read buffer of B?
Hopefully my question makes sense. Thank you for the help.
Is a Scheme input port the same thing as a C FILE* or a python file? Is that the same thing as the unix file descriptor concept? If not, how does an Input Port differe from the others (and why is it it called that way and not just a 'file')?
It's approximately, but only approximately equivalent: ports are objects to and from which you can write or read other objects: bytes or characters (usually).
But the approximation is not terribly close. Ports from which you can read or write characters rather than bytes are, well, ports which handle characters. That means that before they can, for instance, write some octets down on the TCP connection underlying the port, they have to translate those characters into octets using (with luck) some standard encoding. I don't think the mechanism for controlling this encoding & decoding is specified in Scheme, but it must exist. So a port is, at least sometimes, a complicated thing.
(As for why they're called 'ports': well, C made a choice to call communication endpoints 'file descriptors' which makes sense in the context of the Unix 'everything's a file' idea (calling bytes chars was always just a mistake though). But Scheme doesn't come from a Unix/C background so there's really no reason to do that. Given what we call communication endpoints in TCP/IP, 'port' seems quite a good choice of name.)
Is there any way to redirect the content of a tty device (/dev/ttySDIO0) file so that its content can be managed using a file IO operations. I don't prefer to use 'cat' here.
Thank you.
You can use socat which can connect to a tty on one side with a lot of command line options, and the other end can be pretty much anything that accepts read/write or send/receive call. From the above link :
Socat is a command line based utility
that establishes two bidirectional
byte streams and transfers data
between them. Because the streams can
be constructed from a large set of
different types of data sinks and
sources (see address types), and
because lots of address options may be
applied to the streams, socat can be
used for many different purposes.
I'm debugging communications with a serial device, and I need to see all the data flowing both directions.
It seems like this should be easy on Linux, where the serial port is represented by a file. Is there some way that I can do a sort of "bi-directional tee", where I tell my program to connect to a pipe that copies the data to a file and also shuffles it to/from the actual serial port device?
I think I might even know how to write such a beast, but it seems non-trivial, especially to get all of the ioctls passed through for port configuration, etc.
Has anyone already built such a thing? It seems too useful (for people debugging serial device drivers) not to exist already.
strace is very useful for this. You have a visualisation of all ioctl calls, with the corresponding structure decoded. The following options seems particularly useful in your case:
-e read=set
Perform a full hexadecimal and ASCII dump of all the data read from
file descriptors listed in the
specified set. For example, to see all
input activity on file descriptors 3
and 5 use -e read=3,5. Note that this
is independent from the normal tracing
of the read(2) system call which is
controlled by the option -e
trace=read.
-e write=set
Perform a full hexadecimal and ASCII
dump of all the data written to file
descriptors listed in the specified
set. For example, to see all output
activity on file descriptors 3 and 5
use -e write=3,5. Note that this is
independent from the normal tracing of
the write(2) system call which is
controlled by the option -e
trace=write.
I have found pyserial to be quite usable, so if you're into Python it shouldn't be too hard to write such a thing.
A simple method would be to write an application which opened
the master side of a pty and the tty under test. You would then
pass your tty application the slave side of the pty as the 'tty device'.
You would have to monitor the pty attributes with tcgetattr() on the pty
master and call tcsetattr() on the real tty, if the attributes changed.
The rest would be a simple select() on both fd's copying data bi-directionally and copying it to a log.
I looked at a lot of serial sniffers. All of them are based on the idea of making a virtual serial port and sniff data from that port. However, any baud/parity/flow changes will break connection.
So, I wrote my own sniffer :). Most of the serial ports now are just USB-to-serial converters. My sniffer collects data from USB through debugfs, parse it and output to the console. Also any baudrate changes, flow control, line events, and serial errors are also recorded. The project is in the early stage of development and for now, only FTDI is supported.
http://code.google.com/p/uscmon/
Much like #MBR, I was looking into serial sniffers, but the ptys broke the parity check. However, his sniffer was not helping me, as I'm using a CP2102, and not a FT232. So I wrote my own sniffer, by following this, and now I have one that can record file I/O on arbitrary files: I called it tracie.
Ok, I realize this situation is somewhat unusual, but I need to establish a TCP connection (the 3-way handshake) using only raw sockets (in C, in linux) -- i.e. I need to construct the IP headers and TCP headers myself. I'm writing a server (so I have to first respond to the incoming SYN packet), and for whatever reason I can't seem to get it right. Yes, I realize that a SOCK_STREAM will handle this for me, but for reasons I don't want to go into that isn't an option.
The tutorials I've found online on using raw sockets all describe how to build a SYN flooder, but this is somewhat easier than actually establishing a TCP connection, since you don't have to construct a response based on the original packet. I've gotten the SYN flooder examples working, and I can read the incoming SYN packet just fine from the raw socket, but I'm still having trouble creating a valid SYN/ACK response to an incoming SYN from the client.
So, does anyone know a good tutorial on using raw sockets that goes beyond creating a SYN flooder, or does anyone have some code that could do this (using SOCK_RAW, and not SOCK_STREAM)? I would be very grateful.
MarkR is absolutely right -- the problem is that the kernel is sending reset packets in response to the initial packet because it thinks the port is closed. The kernel is beating me to the response and the connection dies. I was using tcpdump to monitor the connection already -- I should have been more observant and noticed that there were TWO replies one of which was a reset that was screwing things up, as well as the response my program created. D'OH!
The solution that seems to work best is to use an iptables rule, as suggested by MarkR, to block the outbound packets. However, there's an easier way to do it than using the mark option, as suggested. I just match whether the reset TCP flag is set. During the course of a normal connection this is unlikely to be needed, and it doesn't really matter to my application if I block all outbound reset packets from the port being used. This effectively blocks the kernel's unwanted response, but not my own packets. If the port my program is listening on is 9999 then the iptables rule looks like this:
iptables -t filter -I OUTPUT -p tcp --sport 9999 --tcp-flags RST RST -j DROP
You want to implement part of a TCP stack in userspace... this is ok, some other apps do this.
One problem you will come across is that the kernel will be sending out (generally negative, unhelpful) replies to incoming packets. This is going to screw up any communication you attempt to initiate.
One way to avoid this is to use an IP address and interface that the kernel does not have its own IP stack using- which is fine but you will need to deal with link-layer stuff (specifically, arp) yourself. That would require a socket lower than IPPROTO_IP, SOCK_RAW - you need a packet socket (I think).
It may also be possible to block the kernel's responses using an iptables rule- but I rather suspect that the rules will apply to your own packets as well somehow, unless you can manage to get them treated differently (perhaps applying a netfilter "mark" to your own packets?)
Read the man pages
socket(7)
ip(7)
packet(7)
Which explain about various options and ioctls which apply to types of sockets.
Of course you'll need a tool like Wireshark to inspect what's going on. You will need several machines to test this, I recommend using vmware (or similar) to reduce the amount of hardware required.
Sorry I can't recommend a specific tutorial.
Good luck.
I realise that this is an old thread, but here's a tutorial that goes beyond the normal SYN flooders: http://www.enderunix.org/docs/en/rawipspoof/
Hope it might be of help to someone.
I can't help you out on any tutorials.
But I can give you some advice on the tools that you could use to assist in debugging.
First off, as bmdhacks has suggested, get yourself a copy of wireshark (or tcpdump - but wireshark is easier to use). Capture a good handshake. Make sure that you save this.
Capture one of your handshakes that fails. Wireshark has quite good packet parsing and error checking, so if there's a straightforward error it will probably tell you.
Next, get yourself a copy of tcpreplay. This should also include a tool called "tcprewrite".
tcprewrite will allow you to split your previously saved capture files into two - one for each side of the handshake.
You can then use tcpreplay to play back one side of the handshake so you have a consistent set of packets to play with.
Then you use wireshark (again) to check your responses.
I don't have a tutorial, but I recently used Wireshark to good effect to debug some raw sockets programming I was doing. If you capture the packets you're sending, wireshark will do a good job of showing you if they're malformed or not. It's useful for comparing to a normal connection too.
There are structures for IP and TCP headers declared in netinet/ip.h & netinet/tcp.h respectively. You may want to look at the other headers in this directory for extra macros & stuff that may be of use.
You send a packet with the SYN flag set and a random sequence number (x). You should receive a SYN+ACK from the other side. This packet will have an acknowledgement number (y) that indicates the next sequence number the other side is expecting to receive as well as another sequence number (z). You send back an ACK packet that has sequence number x+1 and ack number z+1 to complete the connection.
You also need to make sure you calculate appropriate TCP/IP checksums & fill out the remainder of the header for the packets you send. Also, don't forget about things like host & network byte order.
TCP is defined in RFC 793, available here: http://www.faqs.org/rfcs/rfc793.html
Depending on what you're trying to do it may be easier to get existing software to handle the TCP handshaking for you.
One open source IP stack is lwIP (http://savannah.nongnu.org/projects/lwip/) which provides a full tcp/ip stack. It is very possible to get it running in user mode using either SOCK_RAW or pcap.
if you are using raw sockets, if you send using different source mac address to the actual one, linux will ignore the response packet and not send an rst.