How to interact with a subprocess that does not close/complete? - python-3.x

Hi I am trying to write a script that uses Segger's J-Link tool. When the J-Link tool is run it presents a prompt to the user, then commands can issued to it to perform tasks. Here is what I get if I run it from the command prompt:
C:\Program Files (x86)\SEGGER\JLink>jlink
SEGGER J-Link Commander V6.56b (Compiled Dec 6 2019 15:00:15)
DLL version V6.56b, compiled Dec 6 2019 14:58:46
Connecting to J-Link via USB...O.K.
Firmware: Energy Micro EFM32 compiled Apr 20 2016 12:17:23
Hardware version: V1.00
S/N: 440082037
License(s): RDI
VTref=3.297V
Type "connect" to establish a target connection, '?' for help
J-Link>
When I run the following code:
import subprocess
process = subprocess.Popen('C:\\Program Files (x86)\\SEGGER\JLink\\jlink.exe -nonsense', stdout=subprocess.PIPE, universal_newlines=True)
print(process.stdout.read())
I get the following output
SEGGER J-Link Commander V6.56b (Compiled Dec 6 2019 15:00:15)
DLL version V6.56b, compiled Dec 6 2019 14:58:46
Unknown command line option -nonsense.
But if I remove the -nonsense option the process hangs. This make sense as with a nonsense option J-Link exits with an error, in essence closing stdout.
What I want to do in my script is:
Start the J-Link program
Get the initial output from the program (so I can parse it and determine if the script should continue)
Issue commands to J-Link program
Get any responses, parse them and use this in my script
When the script is finished issue qc command which causes J-Link to close.
What am I doing wrong with my subprocess calls that make the process hang, or what I am trying to do will never work and what else can I try?
Edit:
Using a modified version of the answer found here https://stackoverflow.com/a/36477512/9791536
from threading import Thread
def reader(f,buffer):
while True:
line=f.readline()
if line:
buffer.append(line)
else:
break
process = subprocess.Popen("C:\\Program Files (x86)\\SEGGER\\JLink\\jlink.exe -device EFM32GG380F1024 -if SWD -speed 14000 -autoconnect 1", stdout=subprocess.PIPE)
time.sleep(2)
linebuffer=[]
t=Thread(target=reader,args=(process.stdout,linebuffer))
t.daemon=True
t.start()
while True:
if linebuffer:
print(linebuffer.pop(0))
else:
print("nothing here")
time.sleep(1)
I get a stream of "Nothing here" messages. if I replace the command with simple ping that does not return, I get the expected ping results. Seems like Jlink is not playing ball....
process = subprocess.Popen("C:\\windows\system32\ping.exe 127.0.0.1 -t", stdout=subprocess.PIPE)
Edit 2:
In the past I have tried PyLink Module, but didn't work. So tried again now I have more experience with JLink.
Found I need to call set_tif(1) to set the interface to SWD from default of JTAG. So have thrown away subprocess above code.

Related

Why is the serial port still open when killing a subprocess?

I have a programming tool for an ESP32 platform and this programming tool should use the esptool and a task to program the ESP32. This is done by a subprocess:
while(True):
Command = "{}".format(os.getcwd()) + os.path.sep + "esptool.py --port {} --after hard_reset flash_id".format(Port)
print("[DEBUG {}] Run command {}".format(Index, Command))
Proc = subprocess.Popen(Command, shell = True, stdout = subprocess.PIPE)
Output = str()
try:
Output = Proc.communicate(timeout = 10)[0].decode("ascii")
except subprocess.TimeoutExpired:
if(args.debug):
print("[DEBUG {}] Process timeout!".format(Index))
Proc.kill()
# Sometimes the application will stuck here. Use a timeout again to prevent it.
try:
Proc.communicate(timeout = 1)
except subprocess.TimeoutExpired:
pass
# Timeout - jump back and try it again
continue
...
The process is started and a timeout occurs:
[DEBUG 0] Run command ...\app\esptool.py --port COM14 --after hard_reset flash_id
[DEBUG 0] Process timeout!
The application should try the programming process again, but I get SerialException from the esptool.py.
serial.serialutil.SerialException: could not open port 'COM14': PermissionError(13, 'Zugriff verweigert', None, 5)
It looks like the process is still alive. But why? How can I fix it? I´m testing the application with Windows 10, but it also should run under Linux. I´m using Python 3.9.5 64-Bit.

Program runs with sudo and root, not with systemd

I have an application that ran for months on Stretch and Python 2.7 and 3.5, on a Raspberry Pi Model 3B+, however when I moved it to Buster and Python 3.7 I ran into a very strange problem. The error message I get is "error saving session_id: write() argument 1 must be unicode, not str" from a json.dump save of a list.
(code: json.dump(client.user_data, fout))
If I run the script as sudo, to test for the eventual running as root by systemd, it works fine. When I run it as root through sudo su, it also works fine. When I run it through systemd I get the error. For many hours I have been going in circles trying to zoom in on the problem, but now I'm baffled.
At first I though that the issue was with the unicode changes in Python, so I tried to eliminate that as a source of the problem. I even went as far as to add a function to really make sure I'm dealing with unicode, and not strings. (function utfy_dict)
The list I'm using and trying to save looks like this, with some fields edited for pricacy: (taken from a print(data) output":
{u'sessionId': u'555C703C-607x-bla-bla', u'userInfo': {u'username': u'my_email#gmail.com', u'city': u'my_city', u'isActivated': True, u'firstname': u'Paul', u'lastname': u'mylastname', u'userID': my userId, u'zipcode': u'my Zipcode', u'telephone': u'', u'deviceCount': 0, u'streetAddress': u'my street', u'securityQuestion3': u'NotUsed', u'securityQuestion2': u'NotUsed', u'securityQuestion1': u'NotUsed', u'country': u'NL', u'userLanguage': u'nl-NL', u'tenantID': 5, u'latestEulaAccepted': False}}
The relevant pieces:
try:
# save the session-id so we don't need to re-authenticate every polling cycle
print("save session_id")
# here comes the offending piece: an exception is raised
utfy_dict(client.user_data)
print(client.user_data)
print("\n")
with io.open(session_id, "w", encoding="utf-8") as fout:
json.dump(client.user_data, fout)
except Exception as error:# (IOError, ValueError,TypeError):
write_log("error", "error saving session_id: {}".format(error))
return
def utfy_dict(dic):
'''
https://stackoverflow.com/questions/33699343/convert-every-dictionary-value-to-utf-8-dictionary-comprehension
'''
if isinstance(dic,bytes):
return(dic.encode("utf-8")) # the key is this!
elif isinstance(dic,dict):
for key in dic:
dic[key] = utfy_dict(dic[key])
return(dic)
elif isinstance(dic,list):
new_l = []
for e in dic:
new_l.append(utfy_dict(e))
return(new_l)
else:
return(dic)
Systemd service file:
# This service installs a Python script that monitors the CV temperatures.
[Unit]
Description=Installing the cv_mon monitoring script
Requires=basic.target
After=multi-user.target
[Service]
ExecStart=/usr/bin/python /home/pi/cv_app.py
Restart=always
# wait for 10s before the SIGKILL gets send
TimeoutStopSec=10s
# The number of times the service is restarted within a time period can be set
# If that condition is met, the RPi is rebooted
#
StartLimitBurst=4
StartLimitInterval=180s
# actions can be none|reboot|reboot-force|reboot-immidiate
StartLimitAction=none
[Install]
WantedBy=multi-user.target
The get_evo_data function gets called every three minutes. I would like to reiterate that the code ran fine for more than a year on another platform, and it also runs fine as sudo or root, just not under systemd.
Here is an error report: (note the reference to Python 2.7/json ???)
2019-07-31 00:15:30,095 ERROR JSON decode error reading evo_saved_data
2019-07-31 00:15:30,099 ERROR Got exception: write() argument 1 must be unicode, not str
2019-07-31 00:15:30,102 INFO Traceback (most recent call last):
File "/home/pi/cv_app.py", line 717, in main
get_evo_data()
File "/home/pi/cv_app.py", line 314, in get_evo_data
json.dump(client.user_data, fout)
File "/usr/lib/python2.7/json/__init__.py", line 190, in dump
fp.write(chunk)
TypeError: write() argument 1 must be unicode, not str

Raspberry Pi: IR LED works, but irsend does not transmit any IR code

I installed the current lirc package (0.9.0~pre1-1.2) on a Raspian jessie (no pixel) (everything updated and upgraded) and connected to the (lirc default) GPIO ports:
to gpio port 17 - an IR LED via transistor etc
to gpio port 18 - an IR receiver nodule
The receiver part works perfectly:
mode2 command receiving raw data from transmitter
the IR code recognition of previously recorded keys works
However, the IR LED only works only while lirc is not involved:
a shell script can switch the IR LED on and off with no problem
The only thing that doesn't work:
irsend does not make the IR transmitter emit anything, however no error message is shown
So the hardware, especially the IR LED is definitely working, while lirc cannot make the LED emit any configured IR code.
Please note that this seems to be a duplicate of
stackoverflow: irsend is not giving errors, but does not send signal on Raspbian
Unfortunately it is not. The "solution" provided there was placing the data for /etc/modules into the file /etc/modules-load.d/lirc_rpi.conf. I tried that as well, but it makes no difference.
Any help is greatly appreciated!
Configuration data follows - if any other data is required, I'd be happy to add it! TIA!
System and lirc Configuration
Extract fom: /boot/config.txt
dtoverlay=lirc-rpi,gpio_in_pin=18,gpio_out_pin=17,debug=on
Extract of: /etc/modules
lirc_dev
lirc_rpi gpio_in_pin=18 gpio_out_pin=17
(not sure if that is necessary at all, does not make a difference if this is not configured!? Any hint apppreciated)
All active entries in: /etc/lirc/hardware.conf
LIRCD_ARGS="--uinput"
DRIVER="default"
DEVICE="/dev/lirc0"
MODULES="lirc_rpi"
LIRCD_CONF=""
LIRCMD_CONF=""
Some system output
1) The driver is loaded, output of following command right after boot, output of: dmesg | grep lirc
lirc_dev: IR Remote Control driver registered, major 245
lirc_rpi: module is from the staging directory, the quality is unknown, you have been warned.
lirc_rpi: to_irq 178
lirc_rpi: auto-detected active low receiver on GPIO pin 18
lirc_rpi lirc_rpi: lirc_dev: driver lirc_rpi registered at minor = 0
lirc_rpi: driver registered!
input: lircd as /devices/virtual/input/input0
lirc_rpi: Interrupt 178 obtained
2) the service is started and running, output of: systemctl status lirc
? lirc.service - LSB: Starts LIRC daemon.
Loaded: loaded (/etc/init.d/lirc)
Active: active (running) since Mo 2017-06-12 20:04:03 CEST; 2h 58min ago
Process: 377 ExecStart=/etc/init.d/lirc start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/lirc.service
+-437 /usr/sbin/lircd --driver=default --device=/dev/lirc0 --uinput
3) the modules are loaded, output of: lsmod | grep Module;lsmod | grep lirc
Module Size Used by
lirc_rpi 8453 3
lirc_dev 10211 1 lirc_rpi
rc_core 23776 1 lirc_dev
I followed the troubleshooting steps in the (outdated) manual at http://aron.ws/projects/lirc_rpi/
to get some more information.
Output of: cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 0-53, parent: platform/20200000.gpio, pinctrl-bcm2835:
gpio-35 ( |? ) in hi
gpio-47 ( |? ) out lo
I have seen that output also in this case:
raspberrypi.stcakexchange: LIRC won't transmit (irsend: hardware does not support sending)
This user is as irritated by that output as I am - can somebody please tell why gpio-35 and gpio-47 are listed here? shouldn't it be gpio-17 and gpio-18?
Output of: cat /proc/interrupts | grep lirc
178: 875 pinctrl-bcm2835 18 Edge lirc_rpi
This matches the dmesg output on having obtained interrupt 178
Any other dmesg output of lircd, no matter what action, is repeatedly (most likely due to the debug option set) only
lirc_rpi: SET_SEND_CARRIER
lirc_rpi: in init_timing_params, freq=38000 pulse=13157, space=13158
lirc_rpi: SET_SEND_DUTY_CYCLE
lirc_rpi: in init_timing_params, freq=38000 pulse=13157, space=13158
Having restarted testing again after some time for to build up a test copy of the circuit, the problem occurred again. And now, after some more month of much testing, having asked lots of people for help (no one could help), even having purchased and built up a cheap mini USB oscilloscope kit for to examine the hardware further, I finally found the solution.
Long story short: everything in the configuration was correct, and all of the attached hardware was fine. The problem was the testing script - see my remark on
"a shell script can switch the IR LED on and off with no problem"
and as I did not put it in the above description, nobody could have found the solution myself....
The script uses the pseudo files in /sys/class/gpio, see an example here:
raspberry-projects.com: IO pin control from the command line
At the end of the script a command writes to /sys/class/gpio/unexport for cleanup purposes, and this step seems to reset a GPIO port to always end up in the state of being configured for input. As a result LIRC is not longer able to control this GPIO port, since it seems to configure the GPIO port for output only during system boot, and after that always expecting the port to be in that state.
I tracked the problem down to this point by using the gpio utility from the wirinpi package (install with sudo apt-get wiringpi), executing gpio readall and checking for differences.
The time when everything suddenly worked again, I simply may have fogotten about to run my testscript before testing LIRC, which I otherwise always did...
Luckily the problem with the port configuration can easily be fixed without having to reboot the system. Again I use the gpio utility to reset reset the used port for output, where in the below example
the default output port 17 for LIRC is used and
the parameter -g lets the utility use the ordinary GPIO port numbering and not that very different one of the wiringpi package and library
Here is the command, after having executed this last in my test script, LIRC can properly send IR codes again:
gpio -g mode 17 out

PHP exec(myexe) fails in PHP App, but not CLI. Fails Running Under User "apache"

I have a custom program (e.g. myexe) being executed by a web app using PHP's exec() function. It does not fail when run using the PHP CLI nor does myexe fail when run from the command line with me as a user. I have built myexe so that there are no memory issues when profiled using valgrind. myexe is about 26MB in size.
To simplify the situation, I have run myexe on the command line under the user 'apache' and reproduced the failure.
su -s /bin/sh apache -c "/usr/local/bin/myexe parm1 parm2..."
==> Segmentation fault (core dumped)
BUT when I change the user to myself and run the same command above, it works.
su -s /bin/sh mike -c "/usr/local/bin/myexe parm1 parm2..."
==> WORKS
Here's the error from the system log file:
Jul 9 18:26:15 DEVSTN-1 kernel: myexe[27352]: segfault at 7fffa2bf9ff8 ip 0000000000410324 sp 00007fffa2bfa000 error 6 in myexe[400000+5ae000]
Jul 9 18:26:16 DEVSTN-1 abrt[27353]: Saved core dump of pid 27352 (/usr/local/bin/myexe) to /var/spool/abrt/ccpp-2015-07-09-18:26:15-27352 (13631488 bytes)
Jul 9 18:26:16 DEVSTN-1 abrtd: Directory 'ccpp-2015-07-09-18:26:15-27352' creation detected
Jul 9 18:26:17 DEVSTN-1 abrtd: Executable '/usr/local/bin/myexe' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Jul 9 18:26:17 DEVSTN-1 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2015-07-09-18:26:15-27352' exited with 1
Jul 9 18:26:17 DEVSTN-1 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2015-07-09-18:26:15-27352'
My configuration:
CentOS6 2.6.32-504.23.4.el6.x86_64
Apache/2.2.15 (CentOS)
PHP Version 5.3.3
Am I correct with assuming that PHP has nothing to do with the error?
What should I do next?
Correct; PHP has nothing to do with the error. This is a segmentation fault caused by invalid memory access (either overflowing a buffer, or accessing already-freed memory) in myexe. It seems to have saved a core dump to /var/spool/abrt/ccpp-2015-07-09-18:26:15-27352, so, try debugging with GDB:
gdb /usr/local/bin/myexe -c /var/spool/abrt/ccpp-2015-07-09-18:26:15-27352
(gdb) bt
And try to see where the executable is failing. To get useful output, it will need to be compiled with debugging symbols. If it doesn't fail running as root or a different user, or running in an interactive terminal, I'd look for bugs that could be triggered by being unable to open a file, unable to read an expected environment variable, etc. to help isolate your problem.
Running the executable under strace might help figure out what's going on as well.
Found the problem by entering a bash shell user user apache and running the program using gdb.
Turns out myexe was trying to create a directory under the user's home dir (/home/apache) which doesn't exist.
What helped me was knowing how to start a shell under a different user and using gdb.
Here's the command to start a shell under another user (apache):
su -s /bin/bash apache

Is it possible to connect to Mobile Safari remote debugger protocol using python?

I have an HTML5-based application running on iOS and I want to connect to it using the webkit remote debugger protocol 1 that is now supported in iOS 5 2.
I am trying to track down a problem where my javascript application is hard crashing the browser (SEG_FAULT). I would like to get a trace of the application as it executes so I can see what line(s) or network operations may be leading to the issue. My current idea is to write a python application that will connect to the remote debugger and keep stepping through the code and collecting information to a log file while I interact with the application.
I ran into an initial hurdle though that I can't find any examples or documentation about how to connect to the debugger and communicate or even if it is possible.
Does anyone know if this is possible and if so can you point me at some documentation and/or example code?
Based on the code from also below I created a project on github to test out some of the ideas. You can find it here: abierbaum:/python_webkit-remote_debugger
Yes, if you've got the inspector enabled in your UIWebView by following the instructions, it should be possible to connect to it from Python. I played around with it, and figured out how to send and receive commands using a Web Socket. Here's a script for Python 2.7 using websocket-client
import json
import socket
from websocket import WebSocket
ws = WebSocket()
# if ipv6
ws.io_sock = ws.sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
ws.connect("ws://localhost:9999/devtools/page/1")
counter = 0
def send(method, params):
global counter
counter += 1
# separators is important, you'll get "Message should be in JSON format." otherwise
message = json.dumps({"id": counter, "method": method, "params": params}, separators=(',', ':'))
print "> %s" % (message,)
ws.send(message)
def recv():
result = ws.recv()
print "< %s" % (result,)
send('Runtime.evaluate', {'expression': 'alert("hello from python")'})
recv()
This uses the Runtime.evaluate function to show an alert.
I tried running it against MobileSafari running in the simulator, and it worked fine. I noticed two important things:
the remote server is bound to an IPv6 port, and websocket-client didn't connect without the line to override the socket and set the family. Not sure if it would be the same running on a device or in a UIWebView.
it doesn't like spaces around the separators in the JSON.
Here's what it looks like enabling the inspector in MobileSafari using gdb and running the script:
$ ps x | grep MobileSafari
4968 ?? Z 0:00.00 (MobileSafari)
6234 ?? S 0:00.69 /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator5.0.sdk//Applications/MobileSafari.app/MobileSafari
6238 s007 R+ 0:00.00 grep MobileSafari
$ gdb
GNU gdb 6.3.50-20050815 (Apple version gdb-1708) (Thu Nov 3 21:59:02 UTC 2011)
...
(gdb) attach 6234
Attaching to process 6234.
Reading symbols for shared libraries . done
Reading symbols for shared libraries ........................................................................................................................................................ done
0x99798c22 in mach_msg_trap ()
(gdb) p (void *)[WebView _enableRemoteInspector]
$1 = (void *) 0x2ac93ce
(gdb) detach
Detaching from process 6234.
(gdb) quit
$ python debug.py
> {"params":{"expression":"alert(\"hello from python\")"},"id":1,"method":"Runtime.evaluate"}
< {"result":{"result":{"type":"undefined","description":"undefined"}},"id":1}

Resources