More Generally: How can I run a process that runs from the terminal with required keyword arguments using python?
It seems subprocess.Popen is what I want to use.
Here is what I try to run from inside a script:
from subprocess import Popen
Popen(['goodreads-user-scraper', '--user_id 149832357'])
“goodreads-user-scraper” is recognized, but I don’t understand how to pass a keyword argument. I know how to pass an argument, but if I remove “—user_id” I still get same problem.
I have installed goodreads-user-scraper. It works well from the terminal. It is a python package that you can run from the terminal. Repository found here.
Below is how I would ideally like to run the process but from inside python:
pip install goodreads-user-scraper
goodreads-user-scraper --user_id <your id> --output_dir goodreads-data
Result of running from terminal:
Research
In my research I have found these two that seem helpful, but I haven’t been able to adapt the answers to my needs:
Why does passing variables to subprocess.Popen not work despite passing a list of arguments?
How to use subprocess popen Python
Separate all of your arguments like this:
from subprocess import Popen
Popen(['goodreads-user-scraper', '--user_id', '149832357'])
Related
I have tried nearly all solutions in my mind to do this, but the fact is I can't run a python script with modules imported already.
Here's my code for the module cls.py:
import os
def cls():
os.system('cls')
Given below is the code for opening python in cmd:
#echo off
python
pause
What I need is to open the python in the command prompt with the module cls imported. Also, when I try python -m <module> way, it doesn't show any errors, but the program ends.
Any help would be greatly appreciated and thanks in advance.
Saw this question related to mine, but it is not my problem: Run python script that imports modules with batch file
I think what you'r looking for is the interactive mode:
-i
When a script is passed as first argument or the -c option is used, enter interactive mode after executing the script or the command
So just use python -i cls.py in your batch file. This would import your Python file and stay in the Python interpreter prompt. You could then just call cls() because it's already imported.
Alternatively, you could set the environment variable PYTHONSTARTUP in your batch file:
If this is the name of a readable file, the Python commands in that file are executed before the first prompt is displayed in interactive mode. The file is executed in the same namespace where interactive commands are executed so that objects defined or imported in it can be used without qualification in the interactive session.
I'm still a newbie at bash programming, but trying to run a program with little script. Reducing the problem to the error message, I have
cd /full/path/to/program
python3 -m krop
that is the command working when the actual folder is the /full/path/to/program
but if I run the same from root it doesn't work.
cd /another/path
python3 -m /full/path/to/program/krop
/usr/bin/python3: Error while finding module specification for '/full/path/to/program/krop'
(ModuleNotFoundError: No module named 'krop-0')
I tried lot of variants, but always the same output with errors. I do not have a clue of why the library "python3" adds the "-0" at the end of the name of the file.
What should I put to run the program from root?
python -m command expects a module name, similarly to the syntax you would import in a python program. So if your modules lies in ./directory and directory is a valid python module, you can do python -m directory.krop
You can't however index python modules from file system root. You have either to make your bash script run it in the good directory so you make a local import; or you have to package and install your module system-wide to make a global import that would be invoked with python -m krop from anywhere.
More information on packaging and installing modules: https://packaging.python.org/tutorials/packaging-projects/
Problem solved!,
It was a matter of managing the python import paths, as #hiroprotagonist replied. The list that contains all of directories python will use to search for modules, is available in a variable named sys.path.
So, if somebody wants to run a program (a 'library module as a script', according to python help) through python's command, from a directory different from the 'pwd' one, should write in the command line:
export PYTHONPATH='/full/path/to/program/'
python3 -c "import sys; print(sys.path)"
python3 -m krop
The second line is actually to print on screen, but the first one is the only necessary (export PYTHONPATH).
Thank you for the keywords and help!
Ps. May be should be edited the question title to "problem with a python command to run a program from command line on linux" or something like that.
Reference: python --help :)
I want to use Keras on a cluster using Slurm as the job engine.
If I open a terminal and run the following commands, everything works fine:
$python
>>> import tensorflow
>>> import keras
However, if I place import tensorflow and import keras in a Python file that I then call from slurm :
srun [bunch of parameters for my cluster] python mypythonfile.py
Then I get the following error: ImportError: No module named keras.
Is there something specific to do when using Keras in a cluster with Slurm?
I'm reiterating my comment just to show that this question has been answered:
It's common to module load xxxx where xxxx is a different Python installation than the default. You usually stick this in your .bash_profile or a similar file to make sure that you have the Python version you want, always available.
When you submit a job with Slurm, it doesn't call your .bash_profile. It just executes the script. You need to make sure that loading your Python distribution is part of that script.
I also met this problem and solved it recently. As the above answer mentioned, slurm command won't excute .bash_profile and the reason that you can import kears directly in python is the environment setting sentence in the .bash_profile. As a result, I added export PATH="/n/home01/username/miniconda3/bin:$PATH" into my sbatch file and then everything worked.
I am using linux. I am trying to run daemon from function in django views. I want to run shell command from a view in Djangp app. I am using python 2.7. Command needs python2.7 path.
My app will be like plug n play. So on system on which it is going to install may have python installed on different location. So I want to make python path dynamic.
Command will be
usr/bin/python2.7 filename.py --start
On my system path is usr/bin/python2.7.
I found follwing using os.
On python shell I tried following code & I get what I want
import os
getPyPath = os.popen('which python2.7', 'r')
pyPath = getPyPath.read()
pyPath.rstrip()
I got o/p as which is expected as below
usr/bin/python2.7
So now how to get this code is django app view function & run it so that I can get python path in a variable.
I found pythons subprocess module call using which we can run command through shell using shell=True.
So can I get above code running in django view function using subprocess call??
If not what is the other ways to get python path in variable in function django views.
Thanks in advance.
To view the full path to the current Python interpreter, use sys.executable
import sys
print(sys.executable)
I'm experimenting with the threading function in python 3 to get my own pingtesting app/log working, so im following a youtube tutorial
When I've launched a python 3 interpreter, and run:
>>> import threading
>>> print_lock = threading.Lock()
It correctly returns
>>> print_lock
<_thread.lock object at 0x042093C8>
But when I use that piece of code in a script and try to run it as
python scriptName.py
I get an error saying the attribute Lock() doesn't exist
AttributeError: 'module' object has no attribute 'Lock'
How is this possible? I've verified what threading.Lock() returns when running the python interpreter, why isn't it recognized when I try to run it in a script and how can I get this running?
Did you happen to name your module (or another module in the working directory) threading.py? It would get imported ahead of the built-in threading, causing this exact problem.
Trying running:
print(threading.__file__)
in your module, I suspect you'll find it's not the Python built-in.