I have socat creating two serial ports: ttyclient and ttyserver, ttyclient will be called by an application and I have a script listening to ttyserver. The sequence of events I need is
My socat creates the two via /usr/bin/socat -d -d PTY,link=./ttyserver,raw,echo=0 PTY,link=./ttyclient,raw,echo=0
My script writes a few bytes to ttyserver (to be read by ttyclient)
At some later point in time the application runs, reads the bytes from ttyclient and goes to work.
The problem I am having is the bytes are only readable if something is read during step 2. If I run minicom -D ./ttyclient before step 2 I see the bytes being transmitted as expected, but afterwards shows nothing. The data appears to be discarded.
Is this expected behavior? I'm unsure of how to keep the data around so when something reads it is presented with those bytes. Alternatively I'd happy if I had some way to know ttyclient has been opened and send data on that event.
The issue is that I was using Minicom to read ttyclient. For whatever reason Minicom won't display cached data like that... If I just use cat I get my desired behavior.
I've ecountered a problem using a serial GPS/GNSS device on a Raspberry Pi. The device in question is a u-blox GNSS receiver symlinked to /dev/gps.
I try to achieve logging the output data from this device and simultaneously sending correction data to it.
To be more specific, I use RTKLIBs (http://www.rtklib.com/) str2str tool for sending NTRIP/RTCM correction data to the GNSS receiver in order to get better position estimations using DGNSS/RTK.
The receiver's output data will be logged by a python script which is based on the GPS deamon (gpsd).
However, I guess the main issue is related to the serial port control.
When I run the writing process (str2str) first and afterwards any reading process (my python script/gpsd frontends (e.g. cgps) /cat) at the same time, the reading process will output data for a few seconds and freeze then. It doesn't matter which tool I use for reading the data.
I found this question: https://superuser.com/questions/488908/sharing-a-serial-port-between-two-processes. Therefore I made sure that the processes got rw access to the device and even tried running them as superuser. Furthermore I stumbled upon socat and virtual serial ports, but didn't find any use for it. (Virtual Serial Port for Linux)
Is there any way to read data from a serial port with one process and write to it with another? The only solution I know of right now might be to rewrite the read and write process in python using pySerial. This would allow to only have one process accessing the serial device, but would mean plenty of work.
Finally I found a soultion using a construction somehow similar to this: https://serverfault.com/questions/453032/socat-to-share-a-serial-link-between-multiple-processes
A first socat instance (A) gets GNSS correction data from a TCP connection, which is piped to socat B. Socat B manages the connection to the serial device and pipes output data to another socat instance C, which allows other processes such as gpsd to connect and get the receiver's output from TCP port.
In total, this looks like:
socat -d -d -d -u -lpA TCP4:127.0.0.1:10030 - 2>>log.txt |
socat -d -d -d -t3 -lpB - /dev/gps,raw 2>>log.txt|
socat -d -d -d -u -lpC - TCP4-LISTEN:10031,forever,reuseaddr,fork 2>>log.txt
With only one process managing the serial connection, it doesn't block anymore.
I'm working on sending large data files between two Linux computers via a 10 Gigabit Ethernet cable and netcat with a UDP transfer, but seem to be having issues.
After running several tests, I've come to the conclusion that netcat is the issue. I've tested the UDP transfer using [UDT][1], [Tsunami-UDP]2, and a Python UDT transfer as well, and all of which have not had any packet loss issues.
On the server side, we've been doing:
cat "bigfile.txt" | pv | nc -u IP PORT
then on the client side, we've been doing:
nc -u -l PORT > "outputFile.txt"
A few things that we've noticed:
On one of the computers, regardless of whether it's the client or server, it just "hangs". That is to say, even once the transfer is complete, Linux doesn't kill the process and move to the next line in the terminal.
If we run pipe view on the receiving side as well, the incoming data rate is significantly lower than what the sending side thinks it's sending.
Running Wireshark doesn't show any packet loss.
Running the system performance monitor in Linux shows that the incoming data rate (for the receiving side) is the same as the outgoing data rate from the sending side. This is in contrast to what pipe view thinks (see #2)
We're not sure where the issue is with netcat, and if there is a way around it. Any help/insights would be greatly appreciated.
Also, for what it's worth, using netcat with a TCP transfer works fine. And, I do understand that UDP isn't known for reliability, and that packet loss should be expected, but it's the protocol we must use.
Thanks
It could well be that the sending instance is sending the data too fast for the receiving instance. Note that this can occur even if you see no drops on the receiving NIC (as you seem to be saying), because the loss can occur at OS level instead. Your OS could have its UDP buffers overflowing. Run this command:
watch -d "cat /proc/net/snmp | grep -w Udp"
To see if your RcvbufErrors field is non-zero and/or growing while your file transfer is going on.
This answer (How to send only one UDP packet with netcat?) says that nc sends one packet per line. Assuming that's true, this could lead to a significantly higher number of packets than your other transfer mechanisms. Presumably, as #Smeeheey suggested, you're running out of receive buffers on the receiving end.
To cause your sending end to exit, you can add -q 1 to the command line (exit 1 second after seeing end of file).
But there's no way that the the receiving end nc can know when the transfer is complete. This is why these other mechanisms are "protocols" -- they have mechanisms built into them to communicate the bounds of a file. Raw UDP has no concept of end of file.
Tuning the Linux networking stack is a bit complicated, as there are many components to tune to figure out where data is being dropped.
If possible/feasible, I'd recommend that you start by monitoring packet drops throughout the entire network stack. Once you've done that, you can determine where exactly packets are being dropped and then adjust tuning parameters as needed. There are a lot of different files to measure with lots of different fields. I wrote a detailed blog post about monitoring and tuning each component of the Linux networking stack from top to bottom. It's a bit difficult to summarize all the information there, but take a look, I think it can help guide you.
I would like to know if there is any possibility to live capture network packets and save it to a variable for example in python.
I need some information to get from it and not to save it to a file.
I need to capture http packets and get source addres and its content, which should be a html code, to extract only text from it and then do the rest of the job on that information.
There is no reason to save every packet to a file because whole process would be more slower. I was looking for quite a long time for any tool to do this but no success.
Please, if you know any tool that could help me to do this, write about it.
I'm sure you have seen this link about a library to process PCAP files. Now the question is how to acquire in real time without storing in a file.
Probably easiest is to use a fifo
$ mkfifo /tmp/tcpdump.fifo
Now you can capture and feed data into the named fifo
$ sudo tcpdump -s0 -i eth0 -f /tmp/tcpdump.fifo tcp port 80
And in your python program you can open '/tmp/tcpdump.fifo' as the input file as per the instructions in the link.
Alternatively you can try opening '/dev/stdin' in your program and reading the data from there; you could then pipe the PCAP data straight into stdin using the shell and skipping the intermediate named fifo.
$ sudo tcpdump -s0 -i eth0 -f - tcp port 80 | ./youprogram.py
I need to test a serial port application on Linux, however, my test machine only has one serial port.
Is there a way to add a virtual serial port to Linux and test my application by emulating a device through a shell or script?
Note: I cannot remap the port, it hard coded on ttys2 and I need to test the application as it is written.
Complementing the #slonik's answer.
You can test socat to create Virtual Serial Port doing the following procedure (tested on Ubuntu 12.04):
Open a terminal (let's call it Terminal 0) and execute it:
socat -d -d pty,raw,echo=0 pty,raw,echo=0
The code above returns:
2013/11/01 13:47:27 socat[2506] N PTY is /dev/pts/2
2013/11/01 13:47:27 socat[2506] N PTY is /dev/pts/3
2013/11/01 13:47:27 socat[2506] N starting data transfer loop with FDs [3,3] and [5,5]
Open another terminal and write (Terminal 1):
cat < /dev/pts/2
this command's port name can be changed according to the pc. it's depends on the previous output.
2013/11/01 13:47:27 socat[2506] N PTY is /dev/pts/**2**
2013/11/01 13:47:27 socat[2506] N PTY is /dev/pts/**3**
2013/11/01 13:47:27 socat[2506] N starting data transfer loop with FDs
you should use the number available on highlighted area.
Open another terminal and write (Terminal 2):
echo "Test" > /dev/pts/3
Now back to Terminal 1 and you'll see the string "Test".
You can use a pty ("pseudo-teletype", where a serial port is a "real teletype") for this. From one end, open /dev/ptyp5, and then attach your program to /dev/ttyp5; ttyp5 will act just like a serial port, but will send/receive everything it does via /dev/ptyp5.
If you really need it to talk to a file called /dev/ttys2, then simply move your old /dev/ttys2 out of the way and make a symlink from ptyp5 to ttys2.
Of course you can use some number other than ptyp5. Perhaps pick one with a high number to avoid duplicates, since all your login terminals will also be using ptys.
Wikipedia has more about ptys: http://en.wikipedia.org/wiki/Pseudo_terminal
Use socat for this:
For example:
socat PTY,link=/dev/ttyS10 PTY,link=/dev/ttyS11
There is also tty0tty http://sourceforge.net/projects/tty0tty/ which is a real null modem emulator for linux.
It is a simple kernel module - a small source file. I don't know why it only got thumbs down on sourceforge, but it works well for me. The best thing about it is that is also emulates the hardware pins (RTC/CTS DSR/DTR). It even implements TIOCMGET/TIOCMSET and TIOCMIWAIT iotcl commands!
On a recent kernel you may get compilation errors. This is easy to fix. Just insert a few lines at the top of the module/tty0tty.c source (after the includes):
#ifndef init_MUTEX
#define init_MUTEX(x) sema_init((x),1)
#endif
When the module is loaded, it creates 4 pairs of serial ports. The devices are /dev/tnt0 to /dev/tnt7 where tnt0 is connected to tnt1, tnt2 is connected to tnt3, etc.
You may need to fix the file permissions to be able to use the devices.
edit:
I guess I was a little quick with my enthusiasm. While the driver looks promising, it seems unstable. I don't know for sure but I think it crashed a machine in the office I was working on from home. I can't check until I'm back in the office on monday.
The second thing is that TIOCMIWAIT does not work. The code seems to be copied from some "tiny tty" example code. The handling of TIOCMIWAIT seems in place, but it never wakes up because the corresponding call to wake_up_interruptible() is missing.
edit:
The crash in the office really was the driver's fault. There was an initialization missing, and the completely untested TIOCMIWAIT code caused a crash of the machine.
I spent yesterday and today rewriting the driver. There were a lot of issues, but now it works well for me. There's still code missing for hardware flow control managed by the driver, but I don't need it because I'll be managing the pins myself using TIOCMGET/TIOCMSET/TIOCMIWAIT from user mode code.
If anyone is interested in my version of the code, send me a message and I'll send it to you.
You may want to look at Tibbo VSPDL for creating a linux virtual serial port using a Kernel driver -- it seems pretty new, and is available for download right now (beta version). Not sure about the license at this point, or whether they want to make it available commercially only in the future.
There are other commercial alternatives, such as http://www.ttyredirector.com/.
In Open Source, Remserial (GPL) may also do what you want, using Unix PTY's. It transmits the serial data in "raw form" to a network socket; STTY-like setup of terminal parameters must be done when creating the port, changing them later like described in RFC 2217 does not seem to be supported. You should be able to run two remserial instances to create a virtual nullmodem like com0com, except that you'll need to set up port speed etc in advance.
Socat (also GPL) is like an extended variant of Remserial with many many more options, including a "PTY" method for redirecting the PTY to something else, which can be another instance of Socat. For Unit tets, socat is likely nicer than remserial because you can directly cat files into the PTY. See the PTY example on the manpage. A patch exists under "contrib" to provide RFC2217 support for negotiating serial line settings.
Using the links posted in the previous answers, I coded a little example in C++ using a Virtual Serial Port. I pushed the code into GitHub: https://github.com/cymait/virtual-serial-port-example .
The code is pretty self explanatory. First, you create the master process by running ./main master and it will print to stderr the device is using. After that, you invoke ./main slave device, where device is the device printed in the first command.
And that's it. You have a bidirectional link between the two process.
Using this example you can test you the application by sending all kind of data, and see if it works correctly.
Also, you can always symlink the device, so you don't need to re-compile the application you are testing.
Would you be able to use a USB->RS232 adapter? I have a few, and they just use the FTDI driver. Then, you should be able to rename /dev/ttyUSB0 (or whatever gets created) as /dev/ttyS2 .
I can think of three options:
Implement RFC 2217
RFC 2217 covers a com port to TCP/IP standard that allows a client on one system to emulate a serial port to the local programs, while transparently sending and receiving data and control signals to a server on another system which actually has the serial port. Here's a high-level overview.
What you would do is find or implement a client com port driver that would implement the client side of the system on your PC - appearing to be a real serial port but in reality shuttling everything to a server. You might be able to get this driver for free from Digi, Lantronix, etc in support of their real standalone serial port servers.
You would then implement the server side of the connection locally in another program - allowing the client to connect and issuing the data and control commands as needed.
It's probably non trivial, but the RFC is out there, and you might be able to find an open source project that implements one or both sides of the connection.
Modify the linux serial port driver
Alternately, the serial port driver source for Linux is readily available. Take that, gut the hardware control pieces, and have that one driver run two /dev/ttySx ports, as a simple loopback. Then connect your real program to the ttyS2 and your simulator to the other ttySx.
Use two USB<-->Serial cables in a loopback
But the easiest thing to do right now? Spend $40 on two serial port USB devices, wire them together (null modem) and actually have two real serial ports - one for the program you're testing, one for your simulator.
-Adam
$ socat -d -d pty,link=/tmp/vserial1,raw,echo=0 pty,link=/tmp/vserial2,raw,echo=0
Will generate symlinks of /tmp/vserial1 and /tmp/vserial2 for generated virtual serial ports in /dev/pts/*
Resource
Combining all other amazingly useful answers, I found the below command to be VERY useful for testing on different types of Linux distros where there's no guarantee you're going to get the same /dev/pts/#'s every time and/or you need to test multiple psuedo serial devices and connections at once.
parallel 'i="{1}"; socat -d -d pty,raw,echo=0,link=$HOME/pty{1} pty,raw,echo=0,link=$HOME/pty$(($i+1))' ::: $(seq 0 2 3;)
Breaking this down:
parallel runs the same command for each argument supplied to it.
So for example if we run it with the --dryrun flag it gives us:
i="0"; socat -d -d pty,raw,echo=0,link=$HOME/pty0 pty,raw,echo=0,link=$HOME/pty$(($i+1))
i="2"; socat -d -d pty,raw,echo=0,link=$HOME/pty2 pty,raw,echo=0,link=$HOME/pty$(($i+1))
This is due to the $(seq x y z;) at the end, where
x = start #, y = increment by, and z = end # (or # of devices you need to spawn)
parallel 'i="{1}"; echo "make psuedo_devices {1} $(($i+1))"' ::: $(seq 0 2 3;)
Outputs:
make psuedo_devices 0 1
make psuedo_devices 2 3
Gathering all this together the final above command symlinks the proper psuedo devices together regardless of whats in /dev/pts/ to whatever directory supplied to socat via the link flag.
pstree -c -a $PROC_ID gives:
perl /usr/bin/parallel i="{1}"; socat -d -d pty,raw,echo=0,link=$HOME/pty{1} pty,raw,echo=0,link=$HOME/pty$(($i+1)) ::: 0 2
├─bash -c i="0"; socat -d -d pty,raw,echo=0,link=$HOME/pty0 pty,raw,echo=0,link=$HOME/pty$(($i+1))
│ └─socat -d -d pty,raw,echo=0,link=/home/user/pty0 pty,raw,echo=0,link=/home/user/pty1
└─bash -c i="2"; socat -d -d pty,raw,echo=0,link=$HOME/pty2 pty,raw,echo=0,link=$HOME/pty$(($i+1))
└─socat -d -d pty,raw,echo=0,link=/home/user/pty2 pty,raw,echo=0,link=/home/user/pty3
ls -l $HOME/pty* yield:
lrwxrwxrwx 1 user user 10 Sep 7 11:46 /home/user/pty0 -> /dev/pts/4
lrwxrwxrwx 1 user user 10 Sep 7 11:46 /home/user/pty1 -> /dev/pts/6
lrwxrwxrwx 1 user user 10 Sep 7 11:46 /home/user/pty2 -> /dev/pts/7
lrwxrwxrwx 1 user user 10 Sep 7 11:46 /home/user/pty3 -> /dev/pts/8
This was all because I was trying running tests against a platform where I needed to generated a lot of mach-serial connections and to test their input/output via containerization (Docker). Hopefully someone finds it useful.