Program runs with sudo and root, not with systemd - python-3.x

I have an application that ran for months on Stretch and Python 2.7 and 3.5, on a Raspberry Pi Model 3B+, however when I moved it to Buster and Python 3.7 I ran into a very strange problem. The error message I get is "error saving session_id: write() argument 1 must be unicode, not str" from a json.dump save of a list.
(code: json.dump(client.user_data, fout))
If I run the script as sudo, to test for the eventual running as root by systemd, it works fine. When I run it as root through sudo su, it also works fine. When I run it through systemd I get the error. For many hours I have been going in circles trying to zoom in on the problem, but now I'm baffled.
At first I though that the issue was with the unicode changes in Python, so I tried to eliminate that as a source of the problem. I even went as far as to add a function to really make sure I'm dealing with unicode, and not strings. (function utfy_dict)
The list I'm using and trying to save looks like this, with some fields edited for pricacy: (taken from a print(data) output":
{u'sessionId': u'555C703C-607x-bla-bla', u'userInfo': {u'username': u'my_email#gmail.com', u'city': u'my_city', u'isActivated': True, u'firstname': u'Paul', u'lastname': u'mylastname', u'userID': my userId, u'zipcode': u'my Zipcode', u'telephone': u'', u'deviceCount': 0, u'streetAddress': u'my street', u'securityQuestion3': u'NotUsed', u'securityQuestion2': u'NotUsed', u'securityQuestion1': u'NotUsed', u'country': u'NL', u'userLanguage': u'nl-NL', u'tenantID': 5, u'latestEulaAccepted': False}}
The relevant pieces:
try:
# save the session-id so we don't need to re-authenticate every polling cycle
print("save session_id")
# here comes the offending piece: an exception is raised
utfy_dict(client.user_data)
print(client.user_data)
print("\n")
with io.open(session_id, "w", encoding="utf-8") as fout:
json.dump(client.user_data, fout)
except Exception as error:# (IOError, ValueError,TypeError):
write_log("error", "error saving session_id: {}".format(error))
return
def utfy_dict(dic):
'''
https://stackoverflow.com/questions/33699343/convert-every-dictionary-value-to-utf-8-dictionary-comprehension
'''
if isinstance(dic,bytes):
return(dic.encode("utf-8")) # the key is this!
elif isinstance(dic,dict):
for key in dic:
dic[key] = utfy_dict(dic[key])
return(dic)
elif isinstance(dic,list):
new_l = []
for e in dic:
new_l.append(utfy_dict(e))
return(new_l)
else:
return(dic)
Systemd service file:
# This service installs a Python script that monitors the CV temperatures.
[Unit]
Description=Installing the cv_mon monitoring script
Requires=basic.target
After=multi-user.target
[Service]
ExecStart=/usr/bin/python /home/pi/cv_app.py
Restart=always
# wait for 10s before the SIGKILL gets send
TimeoutStopSec=10s
# The number of times the service is restarted within a time period can be set
# If that condition is met, the RPi is rebooted
#
StartLimitBurst=4
StartLimitInterval=180s
# actions can be none|reboot|reboot-force|reboot-immidiate
StartLimitAction=none
[Install]
WantedBy=multi-user.target
The get_evo_data function gets called every three minutes. I would like to reiterate that the code ran fine for more than a year on another platform, and it also runs fine as sudo or root, just not under systemd.
Here is an error report: (note the reference to Python 2.7/json ???)
2019-07-31 00:15:30,095 ERROR JSON decode error reading evo_saved_data
2019-07-31 00:15:30,099 ERROR Got exception: write() argument 1 must be unicode, not str
2019-07-31 00:15:30,102 INFO Traceback (most recent call last):
File "/home/pi/cv_app.py", line 717, in main
get_evo_data()
File "/home/pi/cv_app.py", line 314, in get_evo_data
json.dump(client.user_data, fout)
File "/usr/lib/python2.7/json/__init__.py", line 190, in dump
fp.write(chunk)
TypeError: write() argument 1 must be unicode, not str

Related

How to fix ValueError: I/O operation on closed file for ctx.voice_client.stop() in discord.py on linux

I am trying to run a music bot on a vps running ubuntu. Everything works fine when I run the bot on windows, but when I move it over to my linux vps and use my skip command, it gives me an error: ValueError: I/O operation on closed file. I am not sure how to fix this as it seems to be entirely to do with discord.py's inner workings, as you can see the code simply calls the ctx.voice_client.stop() method.
I have tried looking online and cannot find anyone who mentions this issue, so I believe it to be a problem specific to me, but I cannot figure out why it works on windows and not linux. Here is the command:
#commands.command(name='skip', description="skips to the next song in queue")
async def skip_(self, ctx):
"""Skip the song."""
vc = ctx.voice_client
if not vc or not vc.is_connected():
embed = discord.Embed(title="", description="Not connected to a voice channel")
return await ctx.send(embed=embed)
if not vc.is_playing() and not vc.is_paused():
return await ctx.send("**Nothing is playing**")
vc.stop()
Here is the full error:
Task exception was never retrieved
future: <Task finished name='Task-26' coro=<MusicPlayer.player_loop() done, defined at /home/ubuntu/mystuff/DiscordBots/MusicPlayer/bot.py:144> exception=ValueError('I/O operation on closed file')>
Traceback (most recent call last):
File "/home/ubuntu/mystuff/DiscordBots/MusicPlayer/bot.py", line 177, in player_loop
source.cleanup()
File "/home/ubuntu/.local/lib/python3.10/site-packages/discord/player.py", line 628, in cleanup
self.original.cleanup()
File "/home/ubuntu/.local/lib/python3.10/site-packages/discord/player.py", line 225, in cleanup
self._kill_process()
File "/home/ubuntu/.local/lib/python3.10/site-packages/discord/player.py", line 203, in _kill_process
proc.communicate()
File "/usr/lib/python3.10/subprocess.py", line 1139, in communicate
stdout = self.stdout.read()
ValueError: I/O operation on closed file

Why is the serial port still open when killing a subprocess?

I have a programming tool for an ESP32 platform and this programming tool should use the esptool and a task to program the ESP32. This is done by a subprocess:
while(True):
Command = "{}".format(os.getcwd()) + os.path.sep + "esptool.py --port {} --after hard_reset flash_id".format(Port)
print("[DEBUG {}] Run command {}".format(Index, Command))
Proc = subprocess.Popen(Command, shell = True, stdout = subprocess.PIPE)
Output = str()
try:
Output = Proc.communicate(timeout = 10)[0].decode("ascii")
except subprocess.TimeoutExpired:
if(args.debug):
print("[DEBUG {}] Process timeout!".format(Index))
Proc.kill()
# Sometimes the application will stuck here. Use a timeout again to prevent it.
try:
Proc.communicate(timeout = 1)
except subprocess.TimeoutExpired:
pass
# Timeout - jump back and try it again
continue
...
The process is started and a timeout occurs:
[DEBUG 0] Run command ...\app\esptool.py --port COM14 --after hard_reset flash_id
[DEBUG 0] Process timeout!
The application should try the programming process again, but I get SerialException from the esptool.py.
serial.serialutil.SerialException: could not open port 'COM14': PermissionError(13, 'Zugriff verweigert', None, 5)
It looks like the process is still alive. But why? How can I fix it? I´m testing the application with Windows 10, but it also should run under Linux. I´m using Python 3.9.5 64-Bit.

How to interact with a subprocess that does not close/complete?

Hi I am trying to write a script that uses Segger's J-Link tool. When the J-Link tool is run it presents a prompt to the user, then commands can issued to it to perform tasks. Here is what I get if I run it from the command prompt:
C:\Program Files (x86)\SEGGER\JLink>jlink
SEGGER J-Link Commander V6.56b (Compiled Dec 6 2019 15:00:15)
DLL version V6.56b, compiled Dec 6 2019 14:58:46
Connecting to J-Link via USB...O.K.
Firmware: Energy Micro EFM32 compiled Apr 20 2016 12:17:23
Hardware version: V1.00
S/N: 440082037
License(s): RDI
VTref=3.297V
Type "connect" to establish a target connection, '?' for help
J-Link>
When I run the following code:
import subprocess
process = subprocess.Popen('C:\\Program Files (x86)\\SEGGER\JLink\\jlink.exe -nonsense', stdout=subprocess.PIPE, universal_newlines=True)
print(process.stdout.read())
I get the following output
SEGGER J-Link Commander V6.56b (Compiled Dec 6 2019 15:00:15)
DLL version V6.56b, compiled Dec 6 2019 14:58:46
Unknown command line option -nonsense.
But if I remove the -nonsense option the process hangs. This make sense as with a nonsense option J-Link exits with an error, in essence closing stdout.
What I want to do in my script is:
Start the J-Link program
Get the initial output from the program (so I can parse it and determine if the script should continue)
Issue commands to J-Link program
Get any responses, parse them and use this in my script
When the script is finished issue qc command which causes J-Link to close.
What am I doing wrong with my subprocess calls that make the process hang, or what I am trying to do will never work and what else can I try?
Edit:
Using a modified version of the answer found here https://stackoverflow.com/a/36477512/9791536
from threading import Thread
def reader(f,buffer):
while True:
line=f.readline()
if line:
buffer.append(line)
else:
break
process = subprocess.Popen("C:\\Program Files (x86)\\SEGGER\\JLink\\jlink.exe -device EFM32GG380F1024 -if SWD -speed 14000 -autoconnect 1", stdout=subprocess.PIPE)
time.sleep(2)
linebuffer=[]
t=Thread(target=reader,args=(process.stdout,linebuffer))
t.daemon=True
t.start()
while True:
if linebuffer:
print(linebuffer.pop(0))
else:
print("nothing here")
time.sleep(1)
I get a stream of "Nothing here" messages. if I replace the command with simple ping that does not return, I get the expected ping results. Seems like Jlink is not playing ball....
process = subprocess.Popen("C:\\windows\system32\ping.exe 127.0.0.1 -t", stdout=subprocess.PIPE)
Edit 2:
In the past I have tried PyLink Module, but didn't work. So tried again now I have more experience with JLink.
Found I need to call set_tif(1) to set the interface to SWD from default of JTAG. So have thrown away subprocess above code.

TensorFlow. When running fully_connected_feed.py, app.py occus a 'systemexit' error

I was just learning 'TensorFlow Mechanics 101'. Error occurred when I run fully_connected_feed.py for trainning the MNIST data:
Step 1800: loss = 0.30 (0.002 sec)
Step 1900: loss = 0.44 (0.003 sec)
Training Data Eval:
Num examples: 55000 Num correct: 49180 Precision # 1: 0.8942
Validation Data Eval:
Num examples: 5000 Num correct: 4509 Precision # 1: 0.9018
Test Data Eval:
Num examples: 10000 Num correct: 9023 Precision # 1: 0.9023
An exception has occurred, use %tb to see the full traceback.
SystemExit
D:\software\anaconda\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py:2870:
UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)
After I typed '%tb', it shows:
%tb
Traceback (most recent call last):
File "<ipython-input-1-984b11309266>", line 1, in <module>
runfile('D:/wangjc/pythonTest/TensorFlow/testTensorFlow.py', wdir='D:/wangjc/pythonTest/TensorFlow')
File "D:\software\anaconda\envs\tensorflow\lib\site-packages\spyder\utils\site\sitecustomize.py", line 707, in runfile
execfile(filename, namespace)
File "D:\software\anaconda\envs\tensorflow\lib\site-packages\spyder\utils\site\sitecustomize.py", line 101, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "D:/wangjc/pythonTest/TensorFlow/testTensorFlow.py", line 277, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "D:\software\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
SystemExit
I have found a question like me, but actually it is not the same problem as me. This error message shows different from me:
TypeError: run() got an unexpected keyword argument 'argv'
Also, I installed TensorFlow 1.1.0 by'pip install'.
And, I tried to use several kinds of 'fully_connected_feed.py' code in different TensorFlow version, but other err ( if using lower version ) or the same err occurs.
Some people says the 'SystemExit' error occurs because there is a CMD thread problem. But I don't know whether it is the root and where it is.
Please help me to solve this problem. Thanks!
My ide environment is:
windows 10
Anaconda Python 3.5
TensorFlow 1.1.0
TensorFlow was installed by this code:
(tensorflow) wangjc#wangjc-Inspiron-3668:~$ pip install --ignore-installed --upgrade https://xxxxxxxx.whl
The version shows below:
import tensorflow as tf
tf.VERSION
Out[4]: '1.1.0'
From your traceback:
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
the call to main is wrapped inside a _sys.exit() call that kills the quits the program once it has finished. The message you get comes from running the file inside an iPython iteractive shell. sys.exit() raises a SystemExit exception that normally is used to quit python. iPython's shell, however, captures that exception and shows it to you with a warning. This, however, does not affect your program. Just ignore the message or remove the _sys.exit() wrapping the call to main() in your script.

Crossbar Thruway worker crashes

I have a Crossbar.io server with PHP Thruway workers. Recently, I started getting the following error. It happens about once a day now:
2016-04-17T21:08:12+0000 [Router 9572] Unable to format event {'log_logger': <Logger 'crossbar.router.protocol.WampWebSocketServerProtocol'>, 'log_time': 1460927292.17918, 'log_source': None, 'log_format': 'Traceback (most recent call last):\n File "/usr/local/lib/python2.7/site-packages/autobahn/wamp/websocket.py", line 88, in onMessage\n for msg in self._serializer.unserialize(payload, isBinary):\n File "/usr/local/lib/python2.7/site-packages/autobahn/wamp/serializer.py", line 106, in unserialize\n raise ProtocolError("invalid serialization of WAMP message ({0})".format(e))\nProtocolError: invalid serialization of WAMP message (Expected object or value)\n'}: tuple index out of range
2016-04-17T21:08:15+0000 [Guest 9583] The connected has closed with reason: close
2016-04-17T21:08:19+0000 [Guest 9583] PHP Fatal error: Call to a member function call() on null in /var/www/html/pickupServer/vendor/voryx/thruway/src/Thruway/ClientSession.php on line 106
2016-04-17T21:08:19+0000 [Guest 9583] Fatal error: Call to a member function call() on null in /var/www/html/pickupServer/vendor/voryx/thruway/src/Thruway/ClientSession.php on line 106
2016-04-17T21:08:19+0000 [Controller 9565] Guest worker2 exited with error A process has ended with a probable error condition: process ended with exit code 255.
Does anyone know hot to prevent this?
How do I automatically restart the worker if it fails as in this case?
I solved it in linux with MONIT checking crossbar-controler process and adding the following line:
if children < 3 then restart
3 is the number os child processes that crossbar-controler has on my environment. If any of them exits then crossbar restarts itself and notifies me. You have to check the number of child processes you have running using:
sudo service crossbar status.
This solves the error exit worker but with a cost of restarting the crossbar-controler. I am convinced that must be a crossbar/thruway way of solving the problem.
The ideal way is to try catch all possible errors of php to prevent fatal worker exits.
Thanks

Resources