I would like to understand how variable scoping works in Jupyter notebooks.
When I create a bash notebook with two cells, environment variables that are exported are visible across cell boundaries:
In [1]:
export PATH=$PATH:~/samplepath
In [2]:
echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/myuser/samplepath
But if I create a Python notebook and use cell magics to achieve the same result, variables do not seem to be visible across cell boundaries any more:
In [1]:
%%script bash
export PATH=$PATH:~/samplepath
In [2]:
%%script bash
echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
This behaviour stays the same for different magics (i.e., using an exclamation mark in front of echo instead of the script magic gives the same result).
So, I'd like to understand what the scoping rules are in this case, and how I can make export truly export variables so that they are visible in the entire notebook.
In Jupyter notebook (as in your second example), the shell commands in the notebook are executed in a temporary subshell i.e. every command you write using ! (or under %%bash) runs in a different subprocess.
A good description from this blog post is:
The Jupyter Notebook shell commands are executed in a shell that is a subprocess of the shell running the notebook. Because of this, changes in environmental variables in the child (cell) shell will not be reflected in the parent (notebook) shell.
In[1]:
%%bash
export var="5"
echo $var
5
In[2]:
%%bash
echo $var
No output
This is the very reason why the commands like !cd /path/ can't be used to navigate the file system. Check another blog post on this.
While the case is different with bash notebook, where everything is executed in a single shell.
Solution:
If you want to export a variable to environment shell variables, then you can use %env magic function as follows:
In[3]:
%env var "5"
In[4]:
%env var
In[5]:
%%bash
echo $var
Output of In[4] and In[5]:
'"5"'
Note: But beware, dont't do %env PATH=$PATH:~/samplepath as it'll simply replace the PATH variable thus causing problems. The above solution is recommended for non-system variables only. Nevertheless, you can edit .bashrc to universally change the PATH variable.
Related
This question already has answers here:
Global environment variables in a shell script
(7 answers)
Closed 2 years ago.
I have a seemingly simple bash script to setup my environment:
The first two lines are:
#!/bin/bash
export CVE_ENV_DIR=$PWD
easy, hey? Well, see what happens when I run it, I get the following output:
$ echo $PWD
/work/env
$ ./env.sh
$ echo $CVE_ENV_DIR
$
Why does CVE_ENV_DIR not get set to /work/env? What is happening here? When I type export CVE_ENV_DIR=$PWD manually on the shell, it works as expected...
Child shells cannot affect the environment of their parent. If you want the script to affect the parent's environment, you need to:
source ./env.sh
So what's going on? When you run a bash script, either by bash env.sh or env.sh, you're spawning a program with its own environment, an environment that's divorced from its parent. But, when you run the commands contained in the script at the command line, or using source, there is no spawned environment.
Edit to address #syme's comment. Bash scripts meant to be read using source are often pure configuration, containing only assignments and calculations. It's possible to also make them a little more helpful and self-documenting with a clever she-bang hack like:
#!/bin/echo USAGE: source
# Default configuration file for the Frobnicator package.
FOO=bar
BAR=$(stat /baz)
[[ -f /baz ]] && BAZ=file || BAZ=
export FOO BAR BAZ
Making a bash script meant for configuration look like a configuration script, you help future maintainers. You also help yourself my modularizing your script code into distinct parts, each part with its one unique function.
As a side note, please don't export on the same line as you assign.
I am not much experienced with tch. To simplify the scenario, I created a script and just put "echo $_" there.
if I source the script like this:
source set_env_vars.csh
I get "source set_env_vars.csh"
but if I run the script like:
./set_env_vars.csh
I get a different output 'some paths'
I want to understand what exactly $_ holds and why the outputs are different in two cases.
Thanks
Environmental variables propagate from the shell to the sub-shell only.
source set_env_vars.csh adds the environmental variables to the shell.
./set_env_vars.csh adds the environmental variables to a sub-shell that exists only for the duration of this command. These changes are lost when the sub-shell exits and are not seen by the shell. This is why the outputs are different.
csh:
set a=0
echo "a is $a"
when i do ./my_script.csh output is:
a is
when i do source my_script.csh output is:
a is 0
Why is it so . As i know that ./ execution uses new shell.
That's right, ./my_script.csh starts a new shell, and uses the #! that you should have at the top of the file to select which shell to run (which should be csh in this case).
source my_script.csh runs the script in the current shell.
If the script is incorrectly run in, for example, the bash shell, set a=0 is not the syntax for setting an environment variable in bash, so the code won't work as you expected, because you're using the wrong shell.
Take a look at the #! at the top of the file. Is it correct?
check if variable "a" is set in your current shell:
set | grep '^a='
Remember that once you source script to your current shell,
all it's global variables are there until unset or you exit the current shell.
You may want to start a new shell, source the script, end exit shell to perform valid tests.
I don't know the context of your problem, but you may want to export some key variables to have their copies in every subprocess.
I need to increment an environmental variable by these steps:
envar=1
export envar
sh script_incrementation
echo $envar
where script_incrementation contains something like this:
#! /bin/sh
envar=$[envar+1] #I've tried also other methods of incrementation
export envar
Whatever I do, after exiting the script the variable remains with its initial value 1.
THanks for your time.
A shell script executes in its own shell, so you cannot affect the outer shell unless you source it. See this question for details of that discussion.
Consider the following script, which I will call Foo.sh.
#!/bin/bash
export HELLO=$(($HELLO+1))
Suppose in the outer shell, I define an environmental variable:
export HELLO=1
If I run the script like this, it run inside its own shell and will not affect the parent.
./Foo.sh
However, if I source it, it will just execute the commands in the current shell, and will achieve the desired affect.
. Foo.sh
echo $HELLO # prints 2
Your script can not change the environment of the calling process (shell), it merely inherits it.
So, if you export foo=bar, and then invoke sh (a new process) with your script, the script will see the value of $foo (which is "bar"), and it will be able to change its own copy of it – but that is not going to affect the environment of the parent process (where you exported the variable).
You can simply source your script in the original shell, i.e. run
source increment_script.sh
or
. increment_script.sh
and that will then change the value of the variable.
This is because sourceing a script avoids spawning a new shell (process).
Another trick is to have your script output the changed environment, and then eval that output, for example:
counter=$[counter+1]
echo "counter=$counter"
and then run that as
eval `increment_script.sh`
I (on mac osx) often use
export http_proxy=http://192.168.0.205:1099
to proxy http connection to get a highed download speed. To make things easy， I wrote a shell file named proxy.sh to do this:
#!/bin/sh
export http_proxy=http://192.168.0.205:1099
Before I downlaod, I execute proxy.sh shell command, but I found it did't not come into effect.It lost http_proxy variable in current commnad window(terminal). I must type export command in current terminal,it will come into effect.
So I want to know what's reason for this and a solution? thanks.
Running a shell script "normally" (with proxy.sh for example) results in that running in a sub-process so that it cannot affect the environment of the parent process.
Using . or source will run the shell script in the context of the current shell, so it will be able to affect the environment, using one of the following:
. proxy.sh
source proxy.sh
Another possibility (if you're using bash at least) is to create an alias to do the work for you. You can use something like:
alias faster='export http_proxy=http://192.168.0.205:1099'
so that you can then simply type faster on the command line and it will export that variable (in the context of the current shell).
You could also allow for one-shot settings such as:
alias faster='http_proxy=http://192.168.0.205:1099'
and then use:
faster your_program
which would translate into:
http_proxy=http://192.168.0.205:1099 your_program
That's a bash way to set a variable for just the one invocation of a command.
The export variable will only apply to the script -- if you want it to apply to the shell, you need to use source, and execute the script like so:
. ./proxy.sh
or:
source ./proxy.sh
Note the "." in the first example -- the dot follow by space means the script will apply to the shell.
The reason why your script does not work has been explained by Drakosha & how to make your script work has been explained by Anothony. But with the export in the script you need to source your script each time you open a new terminal. A better solution will be to add the export in .bash_profile or .bashrc
Hope this helps!
When executing a shell script a new shell is launched, the script is executed, and the shell dies. That's why you don't see the variable defined in your shell.
I suggest using an alias for the same purpose.