I’ve used python as my submission script in the past rather than bash etc. In slurm conder I usually instructed the path of the compiler at the top. I tried it in qsub but it didn’t work. Any idea how to make it work?
Code:
#/homes/miranda9/.conda/envs/myenv/lib/python3.7
#PBS -V
#PBS -M me#gmail.com
#PBS -m abe
import sys
for p in sys.path:
print(p)
then I did:
qsub test_qsub.py
error msg:
$ cat test_qsub.py.e381299
/homes/miranda9/.profile: line 2: /opt/intel/compilers_and_libraries_2017/linux/mpi/intel64/bin/mpivars.sh: No such file or directory
/homes/miranda9/.profile: line 3: /opt/intel/compilers_and_libraries_2017/linux/bin/compilervars.sh: No such file or directory
import: unable to open X server `' # error/import.c/ImportImageCommand/369.
/var/spool/pbs/mom_priv/jobs/381299.iam-pbs.SC: line 9: syntax error near unexpected token `print'
/var/spool/pbs/mom_priv/jobs/381299.iam-pbs.SC: line 9: ` print(p)'
after I added ! and made '#!/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.7' at the top I got a new error:
$ cat test_qsub.py.e381301
/homes/miranda9/.profile: line 2: /opt/intel/compilers_and_libraries_2017/linux/mpi/intel64/bin/mpivars.sh: No such file or directory
/homes/miranda9/.profile: line 3: /opt/intel/compilers_and_libraries_2017/linux/bin/compilervars.sh: No such file or directory
-bash: /var/spool/pbs/mom_priv/jobs/381301.iam-pbs.SC: /homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.7: bad interpreter: Permission denied
crossposted:
https://community.openpbs.org/t/how-does-one-use-python-as-a-submission-script-using-the-pbs-scheduler/2271
How does one use python as a submission script using the PBS scheduler?
https://qr.ae/pNFW6P
https://www.reddit.com/r/HPC/comments/issqbu/how_does_one_use_python_as_a_submission_script/
First don't forget the ! at the top.
The trick is to figure out where the binary for python is (with your conda environment for example) and tell PBS at the top for your submission script. I tried using sys.path but that lead to a path to a folder that PBS couldn't use. I had to find out the sys.executable to find it. Then I just put it at the top.
Demo session to find it:
$ python
Python 3.7.7 (default, Mar 26 2020, 15:48:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/homes/miranda9/.conda/envs/myenv/bin/python'
then place it properly with the exclamation mark:
#!/homes/miranda9/.conda/envs/YOURENV/bin/python
#PBS -V
#PBS -M youremail#gmail.com
#PBS -m abe
#PBS -lselect=1:ncpus=112
import sys
import os
for p in sys.path:
print(p)
print(os.environ)
works now!
Related
I have a python script that uses the 'os' library to find the screen size. This script works fine when I run it in SSH but it crashes if I try to run it with crontab.
my python script :
import os
screen_resolution = os.popen("xrandr | grep '*' | awk '{print $1}'").read().strip().split("x")
screen_width = int(screen_resolution[0])
screen_height = int(screen_resolution[1])
print("Screen Resolution : ", screen_width, "x", screen_height)
crontab :
#reboot python /home/pi/Documents/run.py > /home/pi/log.log 2>&1t
or
#reboot env DISPLAY=:0 XAUTHORITY=~/.Xauthority python /home/pi/Documents/run.py > /home/pi/log.log 2>&1
error log :
Can't open display
Traceback (most recent call last):
File "/home/pi/Documents/run.py", line 4, in <module>
screen_width = int(screen_resolution[0])
ValueError: invalid literal for int() with base 10: ''
Is there a way to run this kind of script once the whole system has booted ?
Because I have the impression that by going through cron some of the environment variables have not yet been configured
I tried to pass it variables display and Xauthority in crontab but without success.
I would like to execute this script automatically when you launch by hand via SSH
I think you can more easily determine the resolution of the attached monitor using this command:
/usr/bin/tvservice -s
state 0xa [HDMI DMT (87) RGB full 15:9], 1024x600 # 60.00Hz, progressive
That doesn't require an X11 server to be running, or any authorisation or DISPLAY setting.
You may want to extract just the resolution like this:
/usr/bin/tvservice -s | grep -Po '(?<=,\s)\d+x\d+'
1024x600
Another option is:
/usr/bin/vcgencmd get_lcd_info
1024 600 24
I solved my problem by using the autostart file
nano ~/.config/lxsession/LXDE-pi/autostart
thanks
When I check the process from the bash, it display:
In [42]: !ps
PID TTY TIME CMD
417 ttys000 0:00.49 -bash
7783 ttys000 0:06.50 /Users/me/anaconda3/bin/python /Users/me/anaconda3/bin/ipython
It seems that pid 7783 run two commands simultaneous,
Could please provide any hints help understand it?
It is running only one command but with one argument:
/Users/me/anaconda3/bin/python /Users/me/anaconda3/bin/ipython
^ command ^ argument
Python scripts are not directly executable. An interpreter is required to actually run them. Similarly, in your case the command is the python interpreter, and the argument is the ipython script.
When you directly execute the script, the operating system peeks inside to see if it has a shebang. This is a line starting with #! (actually the byte sequence 0x2321) followed by the path to the program meant to run the file. For example, on my system the ipython script points at the python3.7 interpreter:
$ head -1 $(which ipython3)
#!/usr/local/opt/python/bin/python3.7
Calling the script automatically expands to calling the shebang program with the script. Thus, you never see the actual script being run by itself - only the interpreter running the script.
$ ipython3 -c '!ps' | grep ipython3
5764 ttys004 0:00.37 /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python /Users/miyagi/Library/Python/3.7/bin/ipython3 -c !ps
I want to execute some Linux commands using superuser privileges in Python.
Initially i want to input the password from user and then store the password in variable for future use while executing Linux commands using Superuser privileges.
Below its a sample code, what i intended to do:
import os, sys, getpass**
# To input password from user then use this password while execution of Linux commands using superuser privileges.
password = getpass.getpass()
# Before executing below line it'll prompt to enter the password and here i want to use password variable.
os.system("sudo -s")
# [sudo] password for <username>: # <Password>
After successful execution of above command i want to execute rest of code as superuser privileges.
NOTE: I want to use standard method (i.e stdin, stdout, stderr, pipe etc) for inputing password while execution of os.system("sudo -s") command. I know there are other methods too to deal with this problem.
For Example:
import os
password = "my-password"
command = "echo {0} | sudo -s {1}".format(password, "my-command")
os.system(command)
Please assist me, how can i do it or correct me if my intention is wrong for doing so.
You can do it with pexpect:
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pexpect
>>> terminal = pexpect.spawn('/bin/sh')
Launching the shell
>>> terminal.sendline('sudo aptitude help')
19
Sending the command
>>> terminal.expect('andy:')
0
Expecting [sudo] password for <user>: or just <user>: to appear in the output (terminal will wait for it)
>>> import getpass
>>> password = getpass.getpass()
Password:
Getting the password
>>> terminal.sendline(password)
16
Sending the password to the terminal
>>> result = ''
>>> while True:
... try:
... result += terminal.read_nonblocking(1, 1)
... except pexpect.TIMEOUT:
... break
...
Getting the result
>>> result
'\r\naptitude 0.7.4\r\n\xd0\x98\xd1\x81\xd0\xbf\xd0\xbe\xd0\xbb\xd1\x8c\xd0\xb7\xd0\xbe\xd0\xb2\xd0\xb0\xd0\xbd\xd0\xb8\xd0\xb5: aptitude [-S \xd0\xb8\xd0\xbc\xd1\x8f \xd1\x84\xd0\xb0\xd0\xb9\xd0\xbb\xd0\xb0] [-u|-i]\r\n aptitude [\xd0\xbf\xd0\xb0\xd1\x80\xd0\xb0\xd0\xbc\xd0\xb5\xd1\x82\xd1\x80\xd1\x8b] <\xd0\xb4\xd0\xb5\xd0\xb9\xd1\x81\xd1\x82\xd0\xb2\xd0\xb8\xd0\xb5> \xe2\x80\xa6\r\n \xd0\x94\xd0\xb5\xd0\xb9\xd1\x81\xd1\x82\xd0\xb2\xd0\xb8\xd1\x8f (\xd0\xb5\xd1\x81\xd0\xbb\xd0\xb8 \xd0\xbd\xd0\xb5 \xd0\xb7\...'
(it's Russian)
Pexpect docs here
I downloaded Quokka Python/Flask CMS to a CentOS7 server. Everything works fine with command
sudo python3 manage.py runserver --host 0.0.0.0 --port 80
Then I create a file /etc/init.d/quokkacms. The file contains following code
start() {
echo -n "Starting quokkacms: "
python3 /var/www/quokka/manage.py runserver --host 0.0.0.0 --port 80
touch /var/lock/subsys/quokkacms
return 0
}
stop() {
echo -n "Shutting down quokkacms: "
rm -f /var/lock/subsys/quokkacms
return 0
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
;;
restart)
stop
start
;;
*)
echo "Usage: quokkacms {start|stop|status|restart}"
exit 1
;;
esac
exit $?
But I get error when running sudo service quokkacms start
RuntimeError: Click will abort further execution because Python 3 was
configured to use ASCII as encoding for the environment. Either switch
to Python 2 or consult http://click.pocoo.org/python3/ for
mitigation steps.
It seems to me that it is the bash script. How come I get different results? Also I followed instructions in the link in the error message but still had no luck.
[update] I had already tried the solution provided by Click before I posted this question. Check the results below (i run in root):
[root#webserver quokka]# python3
Python 3.4.3 (default, Jan 26 2016, 02:25:35)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> import codecs
>>> print(locale.getpreferredencoding())
UTF-8
>>> print(codecs.lookup(locale.getpreferredencoding()).name)
utf-8
>>> locale.getdefaultlocale()
('en_US', 'UTF-8')
>>> locale.CODESET
14
>>>
If you are trying to execute tests case you must set the following environment variables each time:
export LC_ALL=en_US.utf-8
export LANG=en_US.utf-8
Doing this each time will resolve the error.
It may also be possible to set this in your IDE run configuration as
LC_ALL=en_US.UTF-8;LANG=en_US.UTF-8
For example see the following setting in PyCharm 2016:
Adding more to the existing solutions:
If you see something like this error in Python 3:
Traceback (most recent call last):
...
RuntimeError: Click will abort further execution because Python 3 was
configured to use ASCII as encoding for the environment. Either switch
to Python 2 or consult http://click.pocoo.org/python3/ for
mitigation steps.
You are dealing with an environment where Python 3 thinks you are restricted to ASCII data. The solution to these problems is different depending on which locale your computer is running in.
For instance, if you have a German Linux machine, you can fix the problem by exporting the locale to de_DE.utf-8:
export LC_ALL=de_DE.utf-8
export LANG=de_DE.utf-8
If you are on a US machine, en_US.utf-8 is the encoding of choice. On some newer Linux systems, you could also try C.UTF-8 as the locale:
export LC_ALL=C.UTF-8
export LANG=C.UTF-8
Taken from the Python 3 Surrogate Handling
I have virtually an identical situation as this question, except the accepted answer does not work for me at all. Making this simple Python script is my second attempt; echoing text and redirecting it to a file doesn't do anything either. I am using the Raspbian linux distro.
pi#raspberrypi ~ $ incrontab -l
/home/pi IN_CREATE,IN_DELETE /home/pi/test.py
pi#raspberrypi ~ $ cat test.py
#! /usr/bin/python3
f = open('test.txt', 'a+')
f.write('success!\n')
f.close()
pi#raspberrypi ~ $ touch abc.123; rm abc.123
pi#raspberrypi ~ $ tail -n 3 /var/log/syslog
May 17 00:17:09 raspberrypi incrond[1799]: (pi) CMD (/home/pi/test.py )
May 17 00:18:36 raspberrypi incrond[1799]: (pi) CMD (/home/pi/test.py )
May 17 00:18:36 raspberrypi incrond[1799]: (pi) CMD (/home/pi/test.py )
pi#raspberrypi ~ $ ls
bin Desktop python_games test.py
Notice the lack of test.txt in the home directory.
I have tested on a standard Debian Wheezy. The problem your script faces come from the fact that the current working directory (CWD) is not what you expect.
Setting an absolute path in your open operation is a way to avoid it:
f = open('/home/pi/test.txt', 'a+')
First, I have fear an infinite recursion if events for test.txt changes trigger again the script, but it seems to be handled by incron.
As stderr is lost when triggered by incron, it is important to test the script manually with ./test.py.
Here is a variation of your script with additional information thanks to $# option:
#! /usr/bin/python3
import sys
import os
f = open('/home/pi/test.txt', 'a+')
f.write('success on ' + sys.argv[1] + ' with CWD=' + os.getcwd() + '\n')
f.close()
Which is registered that way:
$ incrontab -l
/home/pi IN_CREATE,IN_DELETE /home/pi/test.py $#
Now you will see in /home/pi/test.txt
success on /home/pi/ with CWD=/
which explains that your script first tries to write /test.txt and has not required permission on file system to do so.