I am writing a program that handles some data on a server. Throughout the program, many files are made and sent as input into other programs. To do this, I usually make the command string, then run it like so:
cmd = "prog input_file1 input_file2 > outputfile"
os.system(cmd)
When I run the command, however, the programs being called report that they cannot open the files. If I run the python code on my local computer, it is fine. When I loaded it onto the server, it started to fail. I think this is related to issues with permissions, but am not sure how I can fix this. Many of the files, particularly the output files, are being created at run time. The input files have full permissions for all users. Any help or advice would be appreciated!
Cheers!
The python code you list is simple and correct, so the problem is likely not in the two lines of your example. Here are some related areas for you to check out.
Permissions
The user running the python script must have the appropriate permission (read, write, execute). I see from comments that you've already checked this.
What command are you running
If the command is literally typed into your source code like in the example, then you know what command is being run, but if you are generating any part of it (eg. the list of operands, the name of the output file, other parameters, etc), make sure there are no bugs in the portions of your code that generate the command. For example before the call to os.system(cmd) consider including a line like print("About to execute: " + cmd) so you can see exactly what will be run.
Directly invoke the command
If all the above looks good, try to execute the command directly at a terminal on your server. What output do you get then. It's possible that the problem is with the underlying command itself rather than your python code.
Related
I know from experience that if I try to open the same file in Vim in multiple terminals at the same time, I get an error. (Maybe because of temporary files?)
And I know from experience that if I open a text file in Python and read through it, I have to reset the pointer when I'm done.
But I've found that if I run the same Python script in multiple terminals at the same time, I don't get any error; it just successfully runs the script in both. How does this work? Doesn't Python need to read my script from the beginning in order to run it? Is the script copied to a temporary file, or something?
I know from experience that if I try to open the same file in Vim in multiple terminals at the same time, I get an error.
That's not actually true. Vim actually will let you open the same file in multiple terminals at the same time; it's just that it gives you a warning first to let you know that this is happening, so you can abort before you make changes. (It's not safe to modify the file concurrently in two different instances of Vim, because the two instances won't coordinate at all.)
Furthermore, Vim will only give you this warning if you try to open the same file for editing in multiple terminals at the same time. It won't complain if you're just opening the file for reading (using the -R flag).
And I know from experience that if I open a text file in Python and read through it, I have to reset the pointer when I'm done.
That's not exactly true, either. If you make multiple separate calls to open, you'll have multiple separate file objects, and each separately maintains its position in the file. So something like
with open('filename.txt', 'r') as first:
with open('filename.txt', 'r') as second:
print(first.read())
print(second.read())
will print the complete contents of filename.txt twice.
The only reason you'd need to reset the position when you're done reading a file is if you want to use the same file object to read the file again, or if you've opened the file in read/write mode (r+ rather than r) and you now want to switch from reading to writing.
But I've found that if I run the same Python script in multiple terminals at the same time, I don't get any error; it just successfully runs the script in both. How does this work? Doesn't Python need to read my script from the beginning in order to run it? Is the script copied to a temporary file, or something?
As I think should now be clear — there's no problem here. There's no reason that two instances of Python can't both read the same script file at the same time. Linux allows that. (And in fact, if you delete the file, Linux will keep the file on disk until all programs that had it open have either closed it or exited.)
In fact, there's also no reason that two processes can't write to the same file at the same time, though here you have to be very careful to avoid the processes causing problems for each other or corrupting the file.
terminal is just running the command you said it to execute, there is no pointer or anything
you jus
I created a Python project in Pycharm which contains multiple Python files. As of just now, I need to create a run configuration for each Python file in my project, even though they're all the exact same - with the exception of the script.
This seems unnecessary and laborious and I would love to just use one run configuration for multiple Python files.
That said, I'm a novice Python programmer just getting started and so still unfamiliar with large parts of the language.
My Project Files:
My Run Configuration - Used for all Python files:
Some Research Carried Out
I've searched for a solution and explanation to this, but have been unable to find anything. Some of the places I've tried:
JetBrainsTV on youtube (https://www.youtube.com/watch?v=JLfd9LOdu_U)
JetBrains Website (https://www.jetbrains.com/help/pycharm/run-debug-configuration-python.html)
Stack Overflow
I hope there is sufficient detail here, if not I'd be happy to elaborate.
If those files are independent and you have nothing specific to them, then I see two simple ways of running them:
You don't have to manually create a run configuration for every file. You can just Right-Click on the file in the project tree and click "Run "
You can use the Terminal and run them files using the python interpreter as needed.
I was facing a similar situation when I started competitive programming. In my case I wanted to redirect my Test Cases from an input.txt file rather than manually typing the test cases for every run of my code. Using the above solution was not feasible, as I would need to manually change the Script Path and Redirect Input path in the Run Configuration window for every script I was running.
So what I wanted was, one run configuration, that would run all the scripts with Redirect Input path being set to input.txt.
To do that,
I created a main.py file with the following content:
import sys
if __name__ == '__main__':
fname = sys.argv[1]
exec(open(fname).read())
This main.py file is going to run my other python scripts.
Created this run configuration for the main.py file.
Now, every time I needed to run any code, with the code window open, ran this configuration, which actually executed main.py with current file name passed as its argument, which would then also take the inputs redirected from input.txt.
Hope this helps you or anyone trying to run multiple python scripts with a single run configuration in PyCharm.
When running an interactive session, PyCharm thinks of os.getcwd() as my project's directory. However, when I run my script from the command line, PyCharm thinks of os.getcwd() as the directory of the script.
Is there a good workaround for this? Here is what I tried and did not like:
going to Run/Edit Configurations and changing the working directory manually. I did not like this solution, because I will have to do it for every script that I run.
having one line in my code that "fixes" the path for the purposes of interactive sessions and commenting it out before running from command line. This works, but feels wrong.
Is there a way to do this or is it just the way it is supposed to be? Maybe I shouldn't be trying to run random scripts within my project?
Any insight would be greatly appreciated.
Clarification:
By "interactive session" I mean being able to run each line individually in a Python/IPython Console
By "running from command line" I mean creating a script my_script.py and running python path_to_myscript/my_script.py (I actually press the Run button at PyCharm, but I think it's the same).
Other facts that might prove worth mentioning:
I have created a PyCharm project. This contains (among other things) the package Graphs, which contains the module Graph and some .txt files. When I do something within my Graph module (e.g. read a graph from a file), I like to test that things worked as expected. I do this by running a selection of lines (interactively). To read a .txt file, I have to go (using os.path.join()) from the current working directory (the project directory, ...\\project_name) to the module's directory ...\\project_name\\Graphs, where the file is located. However, when I run the whole script via the command line, the command reading the .txt file raises an Error, complaining that no file was found. By looking on the name of the file that was not found, I see that the full file name is something like this:
...\\project_name\\Graphs\\Graphs\\graph1.txt
It seems that this time the current working directory is ...\\project_name\\Graphs\\, and my os.path.join() command actually spoils it.
I user various methods in my python scripts.
set the working directory as first step of your code using os.chdir(some_existing_path)
This would mean all your other paths should be referenced to this, as you hard set the path. You just need to make sure it works from any location and your specifically in your IDE. Obviously, another os.chdir() would change the working directory and os.getcwd() would return the new working directory
set the working directory to __file__ by using os.chdir(os.path.dirname(__file__))
This is actually what I use most, as it is quite reliable, and then I reference all further paths or file operations to this. Or you can simply refer to as os.path.dirname(__file__) in your code without actually changing the working directory
get the working directory using os.getcwd()
And reference all path and file operations to this, knowing it will change based on how the script is launched. Note: do NOT assume that this returns the location of your script, it returns the working directory of the shell !!
[EDIT based on new information]
By "interactive session" I mean being able to run each line
individually in a Python/IPython Console
By running interactively line-by-line in a Python console, the __file__ is not defined, afterall: you are not executing a file. Hence you cannot use os.path.dirname(__file__) you will have to use something like os.chdir(some_known_existing_dir) to reference a path. As a programmer you need to be very aware of working directory and changes to this, your code should reflect that.
By "running from command line" I mean creating a script my_script.py
and running python path_to_myscript/my_script.py (I actually press the
Run button at PyCharm, but I think it's the same).
This, both executing a .py from command line as well as running in your IDE, will populate the __file__, hence you can use os.path.dirname(__file__)
HTH
I am purposely adding another answer to this post, in regards the following:
Other facts that might prove worth mentioning:
I have created a PyCharm project. This contains (among other things)
the package Graphs, which contains the module Graph and some .txt
files. When I do something within my Graph module (e.g. read a graph
from a file), I like to test that things worked as expected. I do this
by running a selection of lines (interactively). To read a .txt file,
I have to go (using os.path.join()) from the current working directory
(the project directory, ...\project_name) to the module's directory
...\project_name\Graphs, where the file is located. However, when I
run the whole script via the command line, the command reading the
.txt file raises an Error, complaining that no file was found. By
looking on the name of the file that was not found, I see that the
full file name is something like this:
...\project_name\Graphs\Graphs\graph1.txt It seems that this time
the current working directory is ...\project_name\Graphs\, and my
os.path.join() command actually spoils it.
I strongly believe that if a python script takes input from any file, that the author of the script needs to cater for this in the script.
What I mean is you as the author need to make sure you know the following regardless of how your script is executed:
What is the working directory
What is the script directory
These two you have no control over when you hand off your script to others, or run it on other peoples machines. The working directory is dependent on how the script is launched. It seems that you run on Windows, so here is an example:
C:\> c:\python\python your_script.py
The working directory is now C:\ if your_script.py is in C:\
C:\some_dir\another_dir\> c:\python\python.exe c:\your_script_dir\your_script.py
The working directory is now C:\some_dir\another_dir
And the above example may even give different results if the SYSTEM PATH variable is set to the path of the location of your_script.py
You need to ensure that your script works even if the user(s) of your script are placing this in various locations on their machines. Some people (and I don't know why) tend to put everything on the Desktop. You need to ensure your script can cope with this, including any spaces in the path name.
Furthermore, if your script is taking input from a file, the you as the author need to ensure that you can cope with changes in working directory, and changes of script directory. There are a few things you may consider:
Have your script input from a known (static) directory, something like C:\python_input\
Have your script input from a known (configurable) directory, use ConfigParser, you can search here on stackoverflow on many posts
Have your script input from a known directory related to the location of the script (using os.path.dirname(__file__))
any other method you may employ to ensure your script can get to the input
Ultimately this is all in your control, and you need to code to ensure it is working.
HTH,
Edwin.
We have put together a perl script that essentially looks at the argument that is being passed to it checks if is creating or modifying a file then it saves that in a mysql database so that it is easily accessible later. Here is the interesting part, how do I make this perl script run before all of the commands typed in the terminal. I need to make this dummy proof so people don't forget to run it.
Sorry I didn't formulate this question properly. What I want to do is prepend to each command such that each command will run like so "./run.pl ls" for example. That way I can track file changes if the command is mv or it creates an out file for example. The script pretty much takes care of that but I just don't know how to run it seamlessly to the user.
I am running ubuntu server with the bash terminal.
Thanks
If I understood correctly you need to execute a function before running every command, something similar to preexec and precmd in zsh.
Unfortunately bash doesn't have a native support for this but you can do it using DEBUG trap.
Here is a sample code applying this method.
This page also provide some useful information.
You can modify the ~/.bashrc file and launch your script there. Do note that each user would (and should) still have the privelege to modify this file, potentially removing the script invocation.
The /etc/bash.bashrc file is system-wide and only changeable by root.
These .bashrcs are executed when a new instance of bash is created (e.g. new terminal).
It is not the same as sh, the system shell, that is dash on Ubuntu systems.
I am developing a Qt application in Linux. I wanted to pass Linux commands to a terminal. That worked but now i also want to get a response from the terminal for this specific command.
For example,
ls -a
As you know this command lists the directories and files of the current working directory. I now want to pass the returned values from the ls call to my application. What is a correct way to do this?
QProcess is the qt class that will let you spawn a process and read the result. There's an example of usage for reading the result of a command on that page.
popen() ， api of linux systerm , return FILE * that you can read it like a file descriptor， may help youp erhaps。
Parsing ls(1) output is dangerous -- make a few files with funny names in a directory and test it out:
touch "one file"
touch "`printf "\x0a\x0a\x0ahello\x0a world"`"
That creates two files in the current working directory. I expect your attempts to parse ls(1) output won't work. This might be alright if you're showing the results to a human, (though a human will be immensely confused if a filename includes output that looks just like ls(1) output!) but if you're trying to present something like an explorer.exe or Finder.app representation of files in the filesystem, this is horribly broken.
Instead, use opendir(3), readdir(3), and closedir(3) to read directory entries yourself. This will be safer, more portable, and (as a side benefit) slightly better performing.