How to investigate which process causes wakeups during laptop sleep-mode in MacOS (or Linux)? - linux

My MacBook spontaneously wakes up from sleep mode with high fan activity.
I want to do a investigate this in RTC or power settings? Or by strace-ing of processes, etc (using some process/kernel magic!).
Hint: It is probably managed by "rtcwake".
I am not even sure if this is a scheduled task, or from a WiFi wakeup, or something else.
I don't want guesses about what usually causes this in Mojave, etc. Instead:
I need to do a systematic investigation on this on my MacOS (Mojave). Linux-related answers are also appreciated.
This is about system standby, sleep-mode, suspended mode. (Note that this is not about standup and wakeup of individual processes. The whole laptop turns on spontaneously.)

Reading the log file is the best way to debug the problem.
So, try this command in your Terminal to fetch the system logs,
this will tell you "wake up" history.
log show --style syslog | fgrep "Wake reason: EC.LidOpen"
To see the wake reason:
For macOS Sierra, Mojave, Catalina, and newer
log show |grep -i "Wake reason"
Or for MacOS El Capitan, Yosemite, Mavericks, and older
syslog |grep -i "Wake reason"
This will look like:
MacBookPro kernel[0] : Wake reason = OHC1
MacBookPro kernel[0] : Wake reason = PWRB
MacBookPro kernel[0] : Wake reason = EHC2
MacBookPro kernel[0] : Wake reason = OHC1
So what do these wake reason codes mean?
OHC: stands for Open Host Controller, is usually USB or Firewire. If you see OHC1 or OHC2 it is almost certainly an external USB keyboard or mouse that has woken up the machine.
EHC: standing for Enhanced Host Controller, is another USB interface, but can also be wireless devices and bluetooth since they are also on the USB bus of a Mac.
USB: a USB device woke the machine up
LID0: this is literally the lid of your MacBook or MacBook Pro when you open the lid the machine wakes up from sleep.
PWRB: PWRB stands for Power Button, which is the physical power button on your Mac
RTC: Real Time Clock Alarm, is generally from wake-on-demand services like when you schedule sleep and wake on a Mac via the Energy Saver control panel. It can also be from launchd setting, user applications, backups, and other scheduled events.
There may be some other codes (like PCI, GEGE, etc) but the above are the ones that most people will encounter in the system logs. Once you find out these codes, you can really narrow down what is causing your Mac to wake up from sleep seemingly at random.
Hope this will help :)

This answer is based on Linux, so it might not apply strictly to Mac.
To determine whether rtcwake is responsible for your MacOS wakeups, you could replace the executable (in my Ubutnu it is /usr/sbin/rtcwake) with a wrapper script that leaves a sign of rtcwake having run, e.g.
$ cd /usr/sbin/rtcwake
$ sudo mv rtcwake rtcwake_orig
and then write script /usr/sbin/rtcwake containing
#!/bin/bash
touch $HOME/rtcwake_ran
/usr/sbin/rtcwake_orig
Variants of the script would depend on your shell.
In particular, in the last line you would possibly run rtcwake in some alternative way, so as to not own the process (nohup / disown).
See https://unix.stackexchange.com/questions/152310/how-to-correctly-start-an-application-from-a-shell
To inspect possible causes of wakeup, you can check various relevant logs, at /var/log.
E.g., syslog*, acpi*.
See also https://unix.stackexchange.com/questions/83036/where-is-the-log-for-acpi-events
Do you have wakeonlan?

Here I am documenting my systematic approach. It is loosely based on, and initiated by, the answer by #vijay-rajpurohit, which is in turn based on comment by #Robert #1431720 . Note that the final result is particular to my MacOS machine, based on the logs shown below. It will be different in your MacOS.
In first attempt, I first checked the logs using: log show --style syslog | grep ... but it is taking too long. I accidentally checked /var/log/wifi.log after exploring the /var/log/ (I am also curious about /var/log/powermanagement/*.asl).
This turned out to be most useful:
cat /var/log/wifi.log|grep -i "Wake reason"
Then found this line: (note the EC. bit)
Thu Apr 23 22:41:32.359 Info: <airportd[219]> _systemWokenByWiFi: System wake reason: <EC.ARPT>, was woken by WiFi
Then googled for EC.ARPT, I found the following commands:
pmset -g log Useful stats about "Total Sleep/Wakes since boot".
pmset -g assertions This turned out to show the full answer to this question:
2020-04-24 02:23:38 +0100
Assertion status system-wide:
BackgroundTask 1
ApplePushServiceTask 0
UserIsActive 1
PreventUserIdleDisplaySleep 0
PreventSystemSleep 0
ExternalMedia 0
PreventUserIdleSystemSleep 0
NetworkClientActive 0
Listed by owning process:
pid 111(hidd): [0x0000200a000986a9] 00:00:00 UserIsActive named: "com.apple.iohideventsystem.queue.tickle.4295010950.3"
pid 85(apsd): [0x0003b830000b90bd] 00:00:10 ApplePushServiceTask named: "com.apple.apsd-waitingformessages-push.apple.com"
Kernel Assertions: 0x100=MAGICWAKE
id=504 level=255 0x100=MAGICWAKE mod=24/04/2020, 01:57 description=en0 owner=en0
Idle sleep preventers: IODisplayWrangler
In short, in a systematic approach, I explored the following keywords based on the logs, and googled each :
EC.ARPT (example link)
iohideventsystem (example link)
MAGICWAKE (example link)
ApplePushServiceTask (see below)
Most informative item emerged from the output of pmset -g assertions. For example ApplePushServiceTask in the following line:
pid 85(apsd): [0x0003b830000b90bd] 00:00:10 ApplePushServiceTask named: "com.apple.apsd-waitingformessages-push.apple.com"
The solution that seems to work in my particular case (not a general solution) was to disable :
/System/Library/LaunchDaemons/com.apple.apsd.plist using launchctl. But this cannot be done until you do a csrutil disable in the safe mode. I don't write instructions here because it need caution and you need to enable it later.
(to be updated)

Related

bluetoothctl scans stop working after some hours

As a followup to this question, I have tried to check if the bug I found with QtBluetooth was connected to bluetoothctl. To do this, I have run this shell script (using expect) in a loop:
#!/usr/bin/expect -f
set prompt "#"
spawn /usr/bin/bluetoothctl
expect $prompt
send "scan on\r\n"
sleep 10
send "scan off\r\n"
send "quit\r"
expect eof
The end goal is to continuously log all the devices around the sensor for an unspecified amount of time (to put a reasonable estimate, consider at least 24 hours of continous operation, with a bluetooth scan performed every 10 seconds).
After around three hours of successful scans, bluetoothctl showed the same behavior as in the linked question (the scan starts, ends, finds no devices despite them being present and discoverable).
Is there anything preventing bluetoothctl to work for this specific task by design? If not, is there any workaround for solving this issue? I'm running the above code on a Linux machine, with Ubuntu 18.10 installed.
Leaving this as an answer for those who might stumble on the same problem and (incorrectly) thinking it to be a programming issue.
This problem is not directly connected with either expect or bluetoothctl. By browsing the event log (dmesg --ctime), I found out that, every time this behavior manifested, this line popped out: Bluetooth: hci0: command <number> tx timeout, with different command numbers. By googling this specific error, it turned out that it might related to the Linux kernel itself, the drivers of the bluetooth device or the power saving options of my Linux machine.

linux kernel consoleblank argument ignored

I'm running an embedded Linux board with a console only (no graphical environment) based on the i.MX6 and a custom Yocto build.
I'm trying to stop the screen from shutting off after 15 minutes of inactivity. I think the correct way to do this is to pass consoleblank=0 to the boot arguments, which I have done. The problem is that when I do
cat /sys/module/kernel/parameters/consoleblank
I get 900. The results of cat /proc/cmdline are:
console=ttymxc0,115200 root=/dev/mmcblk0p2 rootwait rw consoleblank=0
Does anyone know where else that parameter could be set?
Thanks
Marlon
To avoid a console blank after a period of time, there's two things to change:
consoleblank=0 as kernel parameter as you mentioned
disable terminal blanking with: setterm -blank 0 -powerdown 0
The value that you see in proc, I suspect that in the boot process the setterm parameters are set, that will change consoleblank parameter, to be sure about this, you can make a simple test:
setterm -blank 600
cat /sys/module/kernel/parameters/consoleblank
# This must be 600
setterm -blank 0
cat /sys/module/kernel/parameters/consoleblank
# This must be 0
You can see additional info in this question: https://unix.stackexchange.com/questions/8056/disable-screen-blanking-on-text-console
I know this is a super old question, but I recently ran into a very similar problem and discovered that the 15 minute blanking timeout was being caused by Qt. If you're running any Qt programs, that's likely the source of the problem.
There is a function in Qt's source called setTTYCursor. If you look at the code in the linked file, it disables the blanking by setting the timeout to 0 when the Qt app loads, and then when it exits, it changes the blanking to a 15 minute timeout. After it does this, /sys/module/kernel/parameters/consoleblank reports a value of 900, regardless of what you initially supplied with the kernel command line. I spent a lot of time questioning my sanity before figuring out that Qt was changing this behind my back.
You can bypass this bizarre behavior by setting an environment variable prior to launching the Qt application:
export QT_QPA_PRESERVE_CONSOLE_STATE=1

Controlling a USB power supply (on/off) with Linux

Is it possible to turn on/off power supplies from USB manually with Linux?
There's this external USB cooling fan (the kind you use to cool yourself off, not the PC), and it would be nice to be able to control it from the terminal, because I want to position the fan somewhere far away.
I suppose this could also be useful for a variety of other things as well, because there's a lot of USB toys out there. Maybe air purifiers, etc. (I heard they don't really work though).
According to the docs, there were several changes to the USB power management from kernels 2.6.32, which seem to settle in 2.6.38. Now you'll need to wait for the device to become idle, which is governed by the particular device driver. The driver needs to support it, otherwise the device will never reach this state. Unluckily, now the user has no chance to force this. However, if you're lucky and your device can become idle, then to turn this off you need to:
echo "0" > "/sys/bus/usb/devices/usbX/power/autosuspend"
echo "auto" > "/sys/bus/usb/devices/usbX/power/level"
or, for kernels around 2.6.38 and above:
echo "0" > "/sys/bus/usb/devices/usbX/power/autosuspend_delay_ms"
echo "auto" > "/sys/bus/usb/devices/usbX/power/control"
This literally means, go suspend at the moment the device becomes idle.
So unless your fan is something "intelligent" that can be seen as a device and controlled by a driver, you probably won't have much luck on current kernels.
Note. The information in this answer is relevant for the older kernels (up to 2.6.32). See tlwhitec's answer for the information on the newer kernels.
# disable external wake-up; do this only once
echo disabled > /sys/bus/usb/devices/usb1/power/wakeup
echo on > /sys/bus/usb/devices/usb1/power/level # turn on
echo suspend > /sys/bus/usb/devices/usb1/power/level # turn off
(You may need to change usb1 to usb n)
Source: Documentation/usb/power-management.txt.gz
PowerTOP from Intel allows you to toggle devices such as usb peripherals in real-time. These are called 'tunables'.
sudo apt install powertop
sudo powertop
Tab over to 'tunables'.
Scroll down to your device.
Hit enter to toggle power saving mode (Good/Bad)
Note that Bad means the device is always on. Toggling to Good will turn off the device after the preset inactive saving time (default is 2000ms).
See the PowerTOP docs for details on how to make these changes permanent.It generates the config scripts for you (pretty much as described by other posters on this thread).
NOTE: These scripts do not affect USB pin power (which is always on).
These only send the driver protocol to activate and deactivate a device.
If you want to control pin power, you could use either a supported smart USB hub, or better yet a microcontroller.
I have found these solutions that at least work for properly configured Terminus FE 1.1 USB hub chip:
1.To turn off power on all USB ports of a hub, you may unbind the hub from kernel using:
echo "1-4.4.4" > /sys/bus/usb/drivers/usb/unbind
to turn power back on - you may bind it back using
echo "1-4.4.4" > /sys/bus/usb/drivers/usb/bind
2.Switching power at each port individually is trickier: I was able to use hubpower to control each port - but it comes with a downside: hubpower first disconnects the usbdevfs wich causes all of the USB devices to disconect from system, at least on ubuntu:
usb_ioctl.ioctl_code = USBDEVFS_DISCONNECT;
rc = ioctl(fd, USBDEVFS_IOCTL, &usb_ioctl);
With this ioctl disabled I was able to switch off individual port power without detaching all devices - but the power goes back on immediately (probably due to kernel seeing an uninitialized device) which causes USB device just to do a "cold restart" which is what I generally wanted to do. My patched hubpower is here
You could use my tool uhubctl to control USB power per port for compatible USB hubs.
I wanted to do this, and with my USB hardware I couldn't. I wrote a hacky way how to do it here:
http://pintant.cat/2012/05/12/power-off-usb-device/ .
In a short way: I used a USB relay to open/close the VCC of another USB cable...
echo '2-1' |sudo tee /sys/bus/usb/drivers/usb/unbind
works for ubuntu
The reason why folks post questions such as this is due to the dreaded- indeed "EVIL"- USB Auto-Suspend "feature".
Auto suspend winds-down the power to an "idle" USB device and unless the device's driver supports this feature correctly, the device can become uncontactable. So powering a USB port on/off is a symptom of the problem, not the problem in itself.
I'll show you how to GLOBALLY disable auto-suspend, negating the need to manually toggle the USB ports on & off:
Short Answer:
You do NOT need to edit "autosuspend_delay_ms" individually: USB autosuspend can be disabled globally and PERSISTENTLY using the following commands:
sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="/&usbcore.autosuspend=-1 /' /etc/default/grub
update-grub
systemctl reboot
An Ubuntu 18.04 screen-grab follows at the end of the "Long Answer" illustrating how my results were achieved.
Long Answer:
It's true that the USB Power Management Kernel Documentation states autosuspend is to be deprecated and in in its' place "autosuspend_delay_ms" used to disable USB autosuspend:
"In 2.6.38 the "autosuspend" file will be deprecated
and replaced by the "autosuspend_delay_ms" file."
HOWEVER my testing reveals that setting usbcore.autosuspend=-1 in /etc/default/grub as below can be used as a GLOBAL toggle for USB autosuspend functionality- you do NOT need to edit individual "autosuspend_delay_ms" files.
The same document linked above states a value of "0" is ENABLED and a negative value is DISABLED:
power/autosuspend_delay_ms
<snip> 0 means to autosuspend
as soon as the device becomes idle, and negative
values mean never to autosuspend. You can write a
number to the file to change the autosuspend
idle-delay time.
In the annotated Ubuntu 18.04 screen-grab below illustrating how my results were achieved (and reproducible), please remark the default is "0" (enabled) in autosuspend_delay_ms.
Then note that after ONLY setting usbcore.autosuspend=-1 in Grub, these values are now negative (disabled) after reboot. This will save me the bother of editing individual values and can now script disabling USB autosuspend.
Hope this makes disabling USB autosuspend a little easier and more scriptable-
I had a problem when connecting my android phone, I couldn't charge my phone because the power switch on and then off ...
PowerTop let me find this setting and was useful to fix the issue ( auto value was causing issue):
echo 'on' | sudo tee /sys/bus/usb/devices/1-1/power/control
USB 5v power is always on (even when the computer is turned off, on some computers and on some ports.) You will probably need to program an Arduino with some sort of switch, and control it via Serial library from USB plugged in to the computer.
In other words, a combination of this switch tutorial and this tutorial on communicating via Serial libary to Arduino plugged in via USB.
So far I came to the conclusion that you cannot control the power of a USB port. The 5V USB is always provided, and it's up to the device to use it or not. You can check this with a 5V fan or light.
I've tried various methods (disconnect/reconnect/bind/unbind/reset signal). Best so far are bind/unbind as it forces a cold restart of the device (but no power cycle).
I came up with a solution to reset USB devices, ports and controllers in a python script, which supports all of the above methods.
You can find the script at my Github page
Usage:
usb_reset.py -d 8086:1001 --reset-hub
The script uses among others the following solution to reset USB hubs/controllers:
Unbindind a USB port / controller works best via:
echo "myhub" > "/sys/bus/usb/drivers/usb/unbind"
echo "myhub" > "/sys/bus/usb/drivers/usb/bind"
Where myhub is found in /sys/bus/usb/devices/*
Or litteral controllers:
echo "mycontroller" > "/sys/bus/pci/drivers/unbind"
echo "mycontroller" > "/sys/bus/pci/drivers/bind"
Where mycontroller is found in /sys/bus/pci/drivers/[uoex]hci_hcd/*:*

Cygwin top command - See processes for all users

Does anybody know how to see the processes for all users using top command in Cygwin (part of procps library under System).
I know this can be done in *nix but I am struggling in Cygwin. I have tried using pslist but it does not behave in a putty SSH console.
I need to have a solution where I can see a top like dialog using SSH. I do not have any NTLM SSO access to the Win2k3 guest at all so ssh is the only way in.
top only displays Cygwin processes. ps -W will list Windows processes as well.
Manytimes the command "tasklist" gets the job done more effectively. It built into windows, just make sure your System32 folder is part of your bash profile PATH. There is also procps itself. You should also try using mintty for your terminal. You could always try attaching any of these task apps to screen, and or using watch to poll the information.
It seems you can do something like:
wmic process get ProcessId,Name,UserModeTime,KernelModeTime /EVERY:1
The User and Kernel mode times there seem to be expressed in 1/10,000,000th of second.
You should be able to post-process that output to get the CPU-usage per second.
Here using cygwin's perl:
wmic process get ProcessId,Name,UserModeTime,KernelModeTime /EVERY:1 |
perl -lne '
if (/\S/) {
my ($k,$c,$p,$u) = split /\s{2,}/;
$n{"$p\t$c"}=$k+$u;
} else {
my %c;
for my $k (keys %n) {
$c{$k} = $n{$k} - $o{$k} if defined $o{$k}
}
print "$_\t" . $c{$_}/1e5 for (sort {$c{$b}<=>$c{$a}} keys %c)[0..20];
%o = %n; %n = undef; print ""
}'
Outputs something like:
0 System Idle Process 588.12377
2196 sh.exe 107.00075
248 svchost.exe 85.80055
7140 explorer.exe 26.52017
[...]
every second.
Note that if the System Idle Process shows just under 800% on an idle system, that's because your system has 8 CPU cores (well at least 8 threads) as that counts the CPU time of all CPUs.
Also note that the EVERY:1 above is a lie. wmic doesn't seem to give that output every second. More likely, it sleeps roughly 1 second between each report and doesn't compensate for the time it takes to compute the report. So in practice, it will run every 1 second and a bit which means those percentages are not very accurate and slightly overestimated.

Virtual Serial Port for Linux

I need to test a serial port application on Linux, however, my test machine only has one serial port.
Is there a way to add a virtual serial port to Linux and test my application by emulating a device through a shell or script?
Note: I cannot remap the port, it hard coded on ttys2 and I need to test the application as it is written.
Complementing the #slonik's answer.
You can test socat to create Virtual Serial Port doing the following procedure (tested on Ubuntu 12.04):
Open a terminal (let's call it Terminal 0) and execute it:
socat -d -d pty,raw,echo=0 pty,raw,echo=0
The code above returns:
2013/11/01 13:47:27 socat[2506] N PTY is /dev/pts/2
2013/11/01 13:47:27 socat[2506] N PTY is /dev/pts/3
2013/11/01 13:47:27 socat[2506] N starting data transfer loop with FDs [3,3] and [5,5]
Open another terminal and write (Terminal 1):
cat < /dev/pts/2
this command's port name can be changed according to the pc. it's depends on the previous output.
2013/11/01 13:47:27 socat[2506] N PTY is /dev/pts/**2**
2013/11/01 13:47:27 socat[2506] N PTY is /dev/pts/**3**
2013/11/01 13:47:27 socat[2506] N starting data transfer loop with FDs
you should use the number available on highlighted area.
Open another terminal and write (Terminal 2):
echo "Test" > /dev/pts/3
Now back to Terminal 1 and you'll see the string "Test".
You can use a pty ("pseudo-teletype", where a serial port is a "real teletype") for this. From one end, open /dev/ptyp5, and then attach your program to /dev/ttyp5; ttyp5 will act just like a serial port, but will send/receive everything it does via /dev/ptyp5.
If you really need it to talk to a file called /dev/ttys2, then simply move your old /dev/ttys2 out of the way and make a symlink from ptyp5 to ttys2.
Of course you can use some number other than ptyp5. Perhaps pick one with a high number to avoid duplicates, since all your login terminals will also be using ptys.
Wikipedia has more about ptys: http://en.wikipedia.org/wiki/Pseudo_terminal
Use socat for this:
For example:
socat PTY,link=/dev/ttyS10 PTY,link=/dev/ttyS11
There is also tty0tty http://sourceforge.net/projects/tty0tty/ which is a real null modem emulator for linux.
It is a simple kernel module - a small source file. I don't know why it only got thumbs down on sourceforge, but it works well for me. The best thing about it is that is also emulates the hardware pins (RTC/CTS DSR/DTR). It even implements TIOCMGET/TIOCMSET and TIOCMIWAIT iotcl commands!
On a recent kernel you may get compilation errors. This is easy to fix. Just insert a few lines at the top of the module/tty0tty.c source (after the includes):
#ifndef init_MUTEX
#define init_MUTEX(x) sema_init((x),1)
#endif
When the module is loaded, it creates 4 pairs of serial ports. The devices are /dev/tnt0 to /dev/tnt7 where tnt0 is connected to tnt1, tnt2 is connected to tnt3, etc.
You may need to fix the file permissions to be able to use the devices.
edit:
I guess I was a little quick with my enthusiasm. While the driver looks promising, it seems unstable. I don't know for sure but I think it crashed a machine in the office I was working on from home. I can't check until I'm back in the office on monday.
The second thing is that TIOCMIWAIT does not work. The code seems to be copied from some "tiny tty" example code. The handling of TIOCMIWAIT seems in place, but it never wakes up because the corresponding call to wake_up_interruptible() is missing.
edit:
The crash in the office really was the driver's fault. There was an initialization missing, and the completely untested TIOCMIWAIT code caused a crash of the machine.
I spent yesterday and today rewriting the driver. There were a lot of issues, but now it works well for me. There's still code missing for hardware flow control managed by the driver, but I don't need it because I'll be managing the pins myself using TIOCMGET/TIOCMSET/TIOCMIWAIT from user mode code.
If anyone is interested in my version of the code, send me a message and I'll send it to you.
You may want to look at Tibbo VSPDL for creating a linux virtual serial port using a Kernel driver -- it seems pretty new, and is available for download right now (beta version). Not sure about the license at this point, or whether they want to make it available commercially only in the future.
There are other commercial alternatives, such as http://www.ttyredirector.com/.
In Open Source, Remserial (GPL) may also do what you want, using Unix PTY's. It transmits the serial data in "raw form" to a network socket; STTY-like setup of terminal parameters must be done when creating the port, changing them later like described in RFC 2217 does not seem to be supported. You should be able to run two remserial instances to create a virtual nullmodem like com0com, except that you'll need to set up port speed etc in advance.
Socat (also GPL) is like an extended variant of Remserial with many many more options, including a "PTY" method for redirecting the PTY to something else, which can be another instance of Socat. For Unit tets, socat is likely nicer than remserial because you can directly cat files into the PTY. See the PTY example on the manpage. A patch exists under "contrib" to provide RFC2217 support for negotiating serial line settings.
Using the links posted in the previous answers, I coded a little example in C++ using a Virtual Serial Port. I pushed the code into GitHub: https://github.com/cymait/virtual-serial-port-example .
The code is pretty self explanatory. First, you create the master process by running ./main master and it will print to stderr the device is using. After that, you invoke ./main slave device, where device is the device printed in the first command.
And that's it. You have a bidirectional link between the two process.
Using this example you can test you the application by sending all kind of data, and see if it works correctly.
Also, you can always symlink the device, so you don't need to re-compile the application you are testing.
Would you be able to use a USB->RS232 adapter? I have a few, and they just use the FTDI driver. Then, you should be able to rename /dev/ttyUSB0 (or whatever gets created) as /dev/ttyS2 .
I can think of three options:
Implement RFC 2217
RFC 2217 covers a com port to TCP/IP standard that allows a client on one system to emulate a serial port to the local programs, while transparently sending and receiving data and control signals to a server on another system which actually has the serial port. Here's a high-level overview.
What you would do is find or implement a client com port driver that would implement the client side of the system on your PC - appearing to be a real serial port but in reality shuttling everything to a server. You might be able to get this driver for free from Digi, Lantronix, etc in support of their real standalone serial port servers.
You would then implement the server side of the connection locally in another program - allowing the client to connect and issuing the data and control commands as needed.
It's probably non trivial, but the RFC is out there, and you might be able to find an open source project that implements one or both sides of the connection.
Modify the linux serial port driver
Alternately, the serial port driver source for Linux is readily available. Take that, gut the hardware control pieces, and have that one driver run two /dev/ttySx ports, as a simple loopback. Then connect your real program to the ttyS2 and your simulator to the other ttySx.
Use two USB<-->Serial cables in a loopback
But the easiest thing to do right now? Spend $40 on two serial port USB devices, wire them together (null modem) and actually have two real serial ports - one for the program you're testing, one for your simulator.
-Adam
$ socat -d -d pty,link=/tmp/vserial1,raw,echo=0 pty,link=/tmp/vserial2,raw,echo=0
Will generate symlinks of /tmp/vserial1 and /tmp/vserial2 for generated virtual serial ports in /dev/pts/*
Resource
Combining all other amazingly useful answers, I found the below command to be VERY useful for testing on different types of Linux distros where there's no guarantee you're going to get the same /dev/pts/#'s every time and/or you need to test multiple psuedo serial devices and connections at once.
parallel 'i="{1}"; socat -d -d pty,raw,echo=0,link=$HOME/pty{1} pty,raw,echo=0,link=$HOME/pty$(($i+1))' ::: $(seq 0 2 3;)
Breaking this down:
parallel runs the same command for each argument supplied to it.
So for example if we run it with the --dryrun flag it gives us:
i="0"; socat -d -d pty,raw,echo=0,link=$HOME/pty0 pty,raw,echo=0,link=$HOME/pty$(($i+1))
i="2"; socat -d -d pty,raw,echo=0,link=$HOME/pty2 pty,raw,echo=0,link=$HOME/pty$(($i+1))
This is due to the $(seq x y z;) at the end, where
x = start #, y = increment by, and z = end # (or # of devices you need to spawn)
parallel 'i="{1}"; echo "make psuedo_devices {1} $(($i+1))"' ::: $(seq 0 2 3;)
Outputs:
make psuedo_devices 0 1
make psuedo_devices 2 3
Gathering all this together the final above command symlinks the proper psuedo devices together regardless of whats in /dev/pts/ to whatever directory supplied to socat via the link flag.
pstree -c -a $PROC_ID gives:
perl /usr/bin/parallel i="{1}"; socat -d -d pty,raw,echo=0,link=$HOME/pty{1} pty,raw,echo=0,link=$HOME/pty$(($i+1)) ::: 0 2
├─bash -c i="0"; socat -d -d pty,raw,echo=0,link=$HOME/pty0 pty,raw,echo=0,link=$HOME/pty$(($i+1))
│ └─socat -d -d pty,raw,echo=0,link=/home/user/pty0 pty,raw,echo=0,link=/home/user/pty1
└─bash -c i="2"; socat -d -d pty,raw,echo=0,link=$HOME/pty2 pty,raw,echo=0,link=$HOME/pty$(($i+1))
└─socat -d -d pty,raw,echo=0,link=/home/user/pty2 pty,raw,echo=0,link=/home/user/pty3
ls -l $HOME/pty* yield:
lrwxrwxrwx 1 user user 10 Sep 7 11:46 /home/user/pty0 -> /dev/pts/4
lrwxrwxrwx 1 user user 10 Sep 7 11:46 /home/user/pty1 -> /dev/pts/6
lrwxrwxrwx 1 user user 10 Sep 7 11:46 /home/user/pty2 -> /dev/pts/7
lrwxrwxrwx 1 user user 10 Sep 7 11:46 /home/user/pty3 -> /dev/pts/8
This was all because I was trying running tests against a platform where I needed to generated a lot of mach-serial connections and to test their input/output via containerization (Docker). Hopefully someone finds it useful.

Resources