I am trying to run my Expect script from two different locations and it will work with the following Expect executables referenced:
My linux home directory (#!/usr/bin/expect)
A clearcase view on another server (#!/clearlib/vobs/otherdir/bin/expect)
The problem is that I cannot run the script in both places unless I change the reference of the Expect executable location to the first line of the file.
How can I get the correct instance of the Expect executable for the corresponding directory?
If your path is correctly set on both servers, you could use /usr/bin/env:
#!/usr/bin/env expect
That would use the expect as found in the PATH (/usr/bin in one case, /clearlib/vobs/otherdir/bin in the other)
By instead using env as in the example, the interpreter is searched for and located at the time the script is run.
This makes the script more portable, but also increases the risk that the wrong interpreter is selected because it searches for a match in every directory on the executable search path.
It also suffers from the same problem in that the path to the env binary may also be different on a per-machine basis.
And if you have issue with setting the right PATH, then "/usr/bin/env questions regarding shebang line pecularities" can help.
Related
When running an interactive session, PyCharm thinks of os.getcwd() as my project's directory. However, when I run my script from the command line, PyCharm thinks of os.getcwd() as the directory of the script.
Is there a good workaround for this? Here is what I tried and did not like:
going to Run/Edit Configurations and changing the working directory manually. I did not like this solution, because I will have to do it for every script that I run.
having one line in my code that "fixes" the path for the purposes of interactive sessions and commenting it out before running from command line. This works, but feels wrong.
Is there a way to do this or is it just the way it is supposed to be? Maybe I shouldn't be trying to run random scripts within my project?
Any insight would be greatly appreciated.
Clarification:
By "interactive session" I mean being able to run each line individually in a Python/IPython Console
By "running from command line" I mean creating a script my_script.py and running python path_to_myscript/my_script.py (I actually press the Run button at PyCharm, but I think it's the same).
Other facts that might prove worth mentioning:
I have created a PyCharm project. This contains (among other things) the package Graphs, which contains the module Graph and some .txt files. When I do something within my Graph module (e.g. read a graph from a file), I like to test that things worked as expected. I do this by running a selection of lines (interactively). To read a .txt file, I have to go (using os.path.join()) from the current working directory (the project directory, ...\\project_name) to the module's directory ...\\project_name\\Graphs, where the file is located. However, when I run the whole script via the command line, the command reading the .txt file raises an Error, complaining that no file was found. By looking on the name of the file that was not found, I see that the full file name is something like this:
...\\project_name\\Graphs\\Graphs\\graph1.txt
It seems that this time the current working directory is ...\\project_name\\Graphs\\, and my os.path.join() command actually spoils it.
I user various methods in my python scripts.
set the working directory as first step of your code using os.chdir(some_existing_path)
This would mean all your other paths should be referenced to this, as you hard set the path. You just need to make sure it works from any location and your specifically in your IDE. Obviously, another os.chdir() would change the working directory and os.getcwd() would return the new working directory
set the working directory to __file__ by using os.chdir(os.path.dirname(__file__))
This is actually what I use most, as it is quite reliable, and then I reference all further paths or file operations to this. Or you can simply refer to as os.path.dirname(__file__) in your code without actually changing the working directory
get the working directory using os.getcwd()
And reference all path and file operations to this, knowing it will change based on how the script is launched. Note: do NOT assume that this returns the location of your script, it returns the working directory of the shell !!
[EDIT based on new information]
By "interactive session" I mean being able to run each line
individually in a Python/IPython Console
By running interactively line-by-line in a Python console, the __file__ is not defined, afterall: you are not executing a file. Hence you cannot use os.path.dirname(__file__) you will have to use something like os.chdir(some_known_existing_dir) to reference a path. As a programmer you need to be very aware of working directory and changes to this, your code should reflect that.
By "running from command line" I mean creating a script my_script.py
and running python path_to_myscript/my_script.py (I actually press the
Run button at PyCharm, but I think it's the same).
This, both executing a .py from command line as well as running in your IDE, will populate the __file__, hence you can use os.path.dirname(__file__)
HTH
I am purposely adding another answer to this post, in regards the following:
Other facts that might prove worth mentioning:
I have created a PyCharm project. This contains (among other things)
the package Graphs, which contains the module Graph and some .txt
files. When I do something within my Graph module (e.g. read a graph
from a file), I like to test that things worked as expected. I do this
by running a selection of lines (interactively). To read a .txt file,
I have to go (using os.path.join()) from the current working directory
(the project directory, ...\project_name) to the module's directory
...\project_name\Graphs, where the file is located. However, when I
run the whole script via the command line, the command reading the
.txt file raises an Error, complaining that no file was found. By
looking on the name of the file that was not found, I see that the
full file name is something like this:
...\project_name\Graphs\Graphs\graph1.txt It seems that this time
the current working directory is ...\project_name\Graphs\, and my
os.path.join() command actually spoils it.
I strongly believe that if a python script takes input from any file, that the author of the script needs to cater for this in the script.
What I mean is you as the author need to make sure you know the following regardless of how your script is executed:
What is the working directory
What is the script directory
These two you have no control over when you hand off your script to others, or run it on other peoples machines. The working directory is dependent on how the script is launched. It seems that you run on Windows, so here is an example:
C:\> c:\python\python your_script.py
The working directory is now C:\ if your_script.py is in C:\
C:\some_dir\another_dir\> c:\python\python.exe c:\your_script_dir\your_script.py
The working directory is now C:\some_dir\another_dir
And the above example may even give different results if the SYSTEM PATH variable is set to the path of the location of your_script.py
You need to ensure that your script works even if the user(s) of your script are placing this in various locations on their machines. Some people (and I don't know why) tend to put everything on the Desktop. You need to ensure your script can cope with this, including any spaces in the path name.
Furthermore, if your script is taking input from a file, the you as the author need to ensure that you can cope with changes in working directory, and changes of script directory. There are a few things you may consider:
Have your script input from a known (static) directory, something like C:\python_input\
Have your script input from a known (configurable) directory, use ConfigParser, you can search here on stackoverflow on many posts
Have your script input from a known directory related to the location of the script (using os.path.dirname(__file__))
any other method you may employ to ensure your script can get to the input
Ultimately this is all in your control, and you need to code to ensure it is working.
HTH,
Edwin.
This came up in coursework, and I'm stuck:
Many systems have more than one version of a utility program so that users can choose the one they want. Suggest a command to find all the versions of make on a system. What determines which one a user actually gets? How might a user override the defaults?
How would you do that?
How UNIX finds programs
Unix-like systems store their executable programs in various directories for historical reasons.
The directories that are searched when you want to run a command are stored in an environment variable called $PATH, separated by colons (:). To see its contents, type echo "$PATH" in a terminal window. On my system, that shows (split to avoid a long line)
/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:
/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
They're searched in that order. If I want to run make, the system will first check /usr/local/sbin/make (which doesn't exist), then /usr/local/bin/make (also non-existant), then /usr/bin/make (which does exist, so it runs that).
How to figure out which one would run
The program which can be used to look through $PATH to figure out what program would be chosen. Running which make on my system produces the output /usr/bin/make.
Conveniently, which has a -a flag to print all executables that match, not just the first one. (I found this by consulting its manual, by running man which.) So which -a java should tell you where all of the versions of java are.
Changing the defaults
If you like, you can change the contents of the $PATH variable, like you can change any environment variable: If I run PATH="$PATH:/home/anko/bin", the next time the system needs to find a program, it will check through all of what $PATH used to be, plus a directory called bin in my home directory if it couldn't find anything else.
I could also prepend the directory, to make it take precedence over anything else, by doing PATH="/home/anko/bin:$PATH".
Why is bash(.sh) script not executable by default.
I agree that while we touch any file in linux it is created for reading purpose.
But since file name extensions such as sh and csh are for execution purpose.
Won't it be ideal to touch them in an executable mode.
Question might sound redundant to, but i still thought of asking it :)
Ultimately the answer to this question is that that's not what touch was designed to do.
In fact, touch was not even designed to create files; its primary purpose in life is to change file timestamps. It is merely a side effect of that purpose, and the fact that the designers decided to be a little more generous and create the target file if it doesn't already exist (and if you didn't provide the -c option), that it allows you to create files at all.
It should also be mentioned that there are other techniques that create files, such as redirection (echo 'echo '\''I am a script.'\'';' >|script.sh;). The act of file creation is a generic one, and the whole concept of a file is a generic one. A file is just a byte stream; what goes in the byte stream is unspecified at the file abstraction layer. As #AdamGent mentioned, Windows requires certain types of executable files to have certain extensions in order to be executed properly, but even in Windows, you can put executable code in non-executable-extensioned files, and you can put non-executable content in executable-extensioned files. There's no enforcement of file name / file content correspondence at the file layer.
All of that being said, it would often be a convenience if you could easily create a script in Unix that automatically has executable permission set. I've actually written a script to allow me to edit a new file in vim, and then have its permissions set to executable after write-quitting. The reason this potential convenience has not been standardized into a utility is likely related to the concern about security; you don't want people to accidentally make files executable, because that raises the risk of security holes.
You can always write your own program to create a file and make it executable, perhaps based on the extension of the file name.
Another thing that can be added here is that even shell scripts don't always need to be executable. For example, if you write a script that is only intended to be sourced from existing shell processes (via the source or classic . builtins), then the script does not need to be executable at all. Thus, there are cases where the file extension itself does not provide enough information to determine what the appropriate permissions are for the file.
There is nothing in the file name that says a file is even a script. Common practice perhaps says that .sh and .csh are scripts but there is no such rule.
What makes a file an executable script is the magic number in the beginning of the file. The magic number #! (the shebang, which has many other names as well) means the file is a script. For example
#!/bin/bash
When you try to execute the file (it must then also be set to executable in the permissions), the Linux kernel looks at the first two bytes of the file and finds the magic number (#!). It then knows it is supposed to execute whatever comes after the Shebang with the name of the file as argument followed by the rest of the arguments passed.
If you type the following in the shell
blablabla.sh lol1 lol2
The shell recognizes that you are trying to execute something so it invokes the kernel
exec blablabla.sh lol1 lol2
The kernel finds the shebang, and it becomes
exec /bin/bash blablabla.sh lol1 lol2
By exec i mean one of the exec family system calls.
Other fun names for #! include sha-bang, hashbang, pound-bang, hash-exclam and hash-pling
Because the .sh script extension is completely arbitrary in Unix environments. Its only a convention. You can name your script whatever you like so long as it has an executable bit set. This is unlike Windows where I believe its required (.com, .exe, and I think .dll).
touch just changes the timestamp of the file. It again does not care what the file extension of the file is. In fact most tools in Unix do not care about file extension.
On Linux and other Unix-likes, how do I find the path to a specific file that I can use because of the PATH environment variable? For example, if I can use, from the command line:
ls # technically the file name is "ls.exe"
is there a way I can find the path of the ls.exe file explicitly without looking through the PATH variable myself (i.e. maybe have a program search through it but not look myself)?
I understand there are many pitfalls/caveats of doing this, for example a PATH can be a file or even part of a file (I think), plus symlinks, etc. could make it unreliable, but I'm looking for general use cases.
The reason I am asking is I have Windows with msysgit so I have A LOT of folders in my path, and searching every one would be annoying & time-consuming, not to mention harder because of Windows limitations, but I can still presumably use almost anything Linux can use.
$ type ls.exe
ls.exe is /usr/bin/ls.exe
I'm trying to setup a project, storm from git:
https://github.com/nathanmarz/storm/wiki/Setting-up-development-environment
Download a Storm release , unpack it, and put the unpacked bin/ directory on your PATH
My question is: What does PATH mean? What exactly do they want me to do?
Sometimes I see some /bin/path, $PATH, or echo PATH.
Can someone explain the concept of PATH, so I can setup everything easily in the future without just blindly following the instructions?
PATH is a special environment variable in UNIX (and UNIX-like, e.g. GNU/Linux) systems, which is frequently used and manipulated by the shell (though other things can use it, as well).
There's a somewhat terse explanation on wikipedia, but basically it's used to define where to search for executable files (whether binaries, shell scripts, whatever).
You can find out what your current PATH is set to with a simple shell command:
: $; echo $PATH
(Note: the : $; is meant to represent your shell prompt; it may be something very different for you; just know that whatever your prompt is, that's what I'm representing with that string.)
Depending on your system and prior configuration, the value will vary, but a very simple example of the output might be something like:
/usr/bin:/bin:/usr/local/bin
This is a colon(:)-separated list of directories in which to search for executable files (things like ls, et cetera.) In short, when you try to execute a command from your shell (or from within some other program in certain ways), it will search through each of the directories in this list, in order, looking for an executable file of the name you're provided, and run the first one it finds. So that's the concept, per your question.
From there, what this documentation is telling you to do is to add the directory where you've unpacked the software, and in particular its bin subdirectory, into your $PATH variable. How to do this depends a bit on which shell you're using, but for most (Bourne-compatible) shells, you should be able to do something like this, if you're in the directory where that bin directory is:
: $; PATH="$PATH:$PWD/bin"; export PATH
In just about all but an actual Bourne shell, this can be shortened to:
: $; export PATH="$PATH:$PWD/bin"
(I won't bother explaining for CSH-compatible shells (because: I agree with other advice that you don't use them), but something similar can be done in them, as well, if that happens to be your environment of choice for some reason.)
Presumably, though, you'll want to save this to a shell-specific configuration file (could be ~/.profile, ~/.bashrc, ~/.zshrc... depending on your shell), and without reference to $PWD, but rather to whatever it expanded to. One way you might accomplish this would be to do something like this:
: $; echo "export PATH=\"\$PATH:$PWD/bin\""
and then copy/paste the resulting line into the appropriate configuration file.
Of course you could also generate the appropriate command in other ways, especially if your $PWD isn't currently where that bin directory is.
See also:
An article about $PATH (and more)
a related question on superuser.com