I am writing a shell script, to read a file which has key=value pair and set those variables as environment variables. But I have a doubt, if I do source file.txt will that set the variables defined in that file as environment variable or I should read the file line by line and set it using export command ?
Is source command in this case different than export?
When you source the file, the assignments will be set but the variables are not exported unless the allexport option has been set. If you want all the variables to be exported, it is much simpler to use allexport and source the file than it is to read the file and use export explicitly. In other words, you should do:
set -a
. file.txt
(I prefer . because it is more portable than source, but source works just fine in bash.)
Note that exporting a variable does not make it an environment variable. It just makes it an environment variable in any subshell.
source (.) vs export (and also some file lock [flock] stuff at the end):
In short:
source some_script.sh, or the POSIX-compliant equivalent, . some_script.sh, brings variables in from other scripts, while
export my_var="something" pushes variables out to other scripts/processes which are called/started from the current script/process.
Using source some_script.sh or . some_script.sh in a Linux shell script is kind of like using import some_module in Python, or #include <some_header_file.h> in C or C++. It brings variables in from the script being sourced.
Using export some_var="something" is kind of like setting that variable locally, so it is available for the rest of the current script or process, and then also passing it in to any and all sub-scripts or processes you may call from this point onward.
More details:
So, this:
# export `some_var` so that it is set and available in the current script/process,
# as well as in all sub-scripts or processes which are called from the
# current script/process
export some_var="something"
# call other scripts/processes, passing in `some_var` to them automatically
# since it was just exported above!
script1.sh # this script now gets direct access to `some_var`
script2.sh # as does this one
script3.sh # and this one
is as though you had done this:
# set this variable for the current script/process only
some_var="something"
# call other scripts/processes, passing in `some_var` to them **manually**
# so they can use it too
some_var="something" script1.sh # manually pass in `some_var` to this script
some_var="something" script2.sh # manually pass in `some_var` to this script
some_var="something" script3.sh # manually pass in `some_var` to this script
except that the first version above, where we called export some_var="something" actually has a recursive passing or exporting of variables to sub-processes, so if we call script1.sh from inside our current script/process, then script1.sh will get the exported variables from our current script, and if script1.sh calls script5.sh, and script5.sh calls script10.sh, then both of those scripts as well will get the exported variables automatically. This is in contrast to the manual case above where only those scripts called explicitly with manually-set variables as the scripts are called will get them, so sub-scripts will NOT automatically get any variables from their calling scripts!
How to "un-export" a variable:
Note that once you've exported a variable, calling unset on it will "unexport it", like this:
# set and export `some_var` so that sub-processes will receive it
export some_var="something"
script1.sh # this script automatically receives `some_var`
# unset and un-export `some_var` so that sub-processes will no longer receive it
unset some_var
script1.sh # this script does NOT automatically receive `some_var`
In summary:
source or . imports.
export exports.
unset unexports.
Example:
Create this script:
source_and_export.sh:
#!/bin/bash
echo "var1 = $var1"
var2="world"
Then mark it executable:
chmod +x source_and_export.sh
Now here is me running some commands at the terminal to test the source (.) and export commands with this script. Type in the command you see after the lines beginning with $ (not including the comments). The other lines are the output. Run the commands sequentially, one command at a time:
$ echo "$var1" # var1 contains nothing locally
$ var1="hello" # set var1 to something in the current process only
$ ./source_and_export.sh # call a sub-process
var1 = # the sub-process can't see what I just set var1 to
$ export var1 # **export** var1 so sub-processes will receive it
$ ./source_and_export.sh # call a sub-process
var1 = hello # now the sub-process sees what I previously set var1 to
$ echo "$var1 $var2" # but I can't see var2 from the subprocess/subscript
hello
$ . ./source_and_export.sh # **source** the sub-script to _import_ its var2 into the current process
var1 = hello
$ echo "$var1 $var2" # now I CAN see what the subprocess set var2 to because I **sourced it!**
hello world # BOTH var1 from the current process and var2 from the sub-process print in the current process!
$ unset var1 # unexport (`unset`) var1
$ echo "$var1" # var1 is now NOT set in the current process
$ ./source_and_export.sh # and the sub-process doesn't receive it either
var1 =
$ var1="hey" # set var1 again in the current process
$ . ./source_and_export.sh # if I **source** the script, it runs in the current process, so it CAN see var1 from the current process!
var1 = hey # notice it prints
$ ./source_and_export.sh # but if I run the script as a sub-process, it can NOT see var1 now because it was `unset` (unexported)
var1 = # above and has NOT been `export`ed again since then!
$
Using files as global variables between processes
Sometimes, when writing scripts to launch programs and things especially, I have come across cases where export doesn't seem to work right. In these cases, sometimes one must resort to using files themselves as global variables to pass information from one program to another. Here is how that can be done. In this example, the existence of the file "~/temp/.do_something" functions as an inter-process boolean variable:
# In program A, if the file "~/temp/.do_something" does NOT exist,
# then create it
mkdir -p ~/temp
if [ ! -f ~/temp/.do_something ]; then
touch ~/temp/.do_something # create the file
fi
# In program B, check to see if the file exists, and act accordingly
mkdir -p ~/temp
DO_SOMETHING="false"
if [ -f ~/temp/.do_something ]; then
DO_SOMETHING="true"
fi
if [ "$DO_SOMETHING" == "true" ] && [ "$SOME_OTHER_VAR" == "whatever" ]; then
# remove this global file "variable" so we don't act on it again
# until "program A" is called again and re-creates the file
rm ~/temp/.do_something
do_something
else
do_something_else
fi
Simply checking for the existence of a file, as shown above, works great for globally passing around boolean conditions between programs and processes. However, if you need to pass around more complicated variables, such as strings or numbers, you may need to do this by writing these values into the file. In such cases, you should use the file lock function, flock, to properly ensure inter-process synchronization. It is a type of process-safe (ie: "inter-process") mutex primitive. You can read about it here:
The shell script flock command: https://man7.org/linux/man-pages/man1/flock.1.html. See also man flock or man 1 flock.
The Linux library C command: https://man7.org/linux/man-pages/man2/flock.2.html. See also man 2 flock. You must #include <sys/file.h> in your C file to use this function.
References:
https://askubuntu.com/questions/862236/source-vs-export-vs-export-ld-library-path/862256#862256
My own experimentation and testing
I'll be adding the above example to my project on GitHub here, under the bash folder: https://github.com/ElectricRCAircraftGuy/eRCaGuy_hello_world
Related
I'm making a thing which has to source
a POSIX shell script (NOT BASH) as its configuration
file, but it should only have definitions of variables,
functions and comments, nothing else like running something in
global scope, so I wonder, is there a way to
stop users from being able to run any code globally,
only in functions and stuff?
I mean:
main.sh
# ...
# . in POSIX sh has the same meaning as "source" in bash
. /etc/something/config.sh
echo "$(get_hello), $ME"
# ...
/etc/something/config.sh
#!/usr/bin/env sh
# This one is fine
export ME='Some user'
hello='hi :)'
# This is fine too
get_hello() {
echo '$hello'
}
# This should error or be ignored
echo 'I should not be executed/should throw an error'
Any way to do this?
In my current project I have the problem of overlong commands in the slurm multiprog conf (it has a limit of 256 chars).
To circumvent this, I'd like to define variables (paths, userdata, filenames) in the batchfile and use them in the multiprog config file.
Aunt google and the rather spartan documentation didn't give me hints. I looked into using environment variables, but haven't found a way to set my own.
Any hints?
The command line in the multiprog configuration file is exec'ed rather than parsed with Bash so you need to invoke the Bash shell explicitly.
For instance:
$ export VAR=VALUE
I set a VAR var in the environment with value VALUE and use the following conf file to illustrate:
$ cat multi.conf
0 echo $VAR
1 bash -c 'echo $VAR'
Task 0 will simply be exec'ed while task 1 will first be parsed by Bash. The result:
$ srun -n2 -l --multi-prog multi.conf
0: $VAR
1: VALUE
Taks 0 echoes the variable name while task 1 echoes the variable's value. But beware that you need to spend 10 additional chars.
I have created a perl script which invokes two bash script. First script will set a envirnomental variable and the second will echo the environmental variable. I have given the contents of the files bellow
# perlscript.pl
print `. setnameenv.sh`;
print `. getnameenv.sh`;
# setnameenv.sh
export my_msg='hello world!'
# getnameenv.sh
echo $my_msg
now when I run the perl script perl perlscript.pl I am expecting the 'hello world' to be printed on the screen but actually I don't see any output. I there any way to do this without modifying the bash scripts?
You can embed perl into bash script,
#!/bin/bash
. setnameenv.sh
exec perl -x "$0" "$#"
#!perl
# your script below
print `. getnameenv.sh`;
From perldoc
-x
-xdirectory
tells Perl that the program is embedded in a larger chunk of unrelated text, such as in a mail message. Leading garbage will be discarded until the first line that starts with #! and contains the string "perl". Any meaningful switches on that line will be applied.
You spawn a shell, execute some commands to change its environment, then exit the shell. You never used the environment variable you created before exiting the shell. If you want a perl to see it, you're going to have to launch Perl from that shell.
. setnameenv.sh ; perlscript.pl
If you can't change how perlscript.pl is launched, you have a couple of options, none of which are that friendly. One of the options is to bootstrap.
BEGIN {
if (!length($ENV{my_msg})) {
require String::ShellQuote;
my $cmd = join(' ; ',
'. setnameenv.sh',
String::ShellQuote::shell_quote($^X, $0, #ARGV),
);
exec($cmd)
or die $!;
}
}
This can now be done in Perl with the Env::Modify module.
use Env::Modify qw(source);
source("setnameenv.sh");
# env settings from setnameenv.sh are now available to Perl
# and to the following system call
print `. getenvname.sh`; # or source again, like source("getenvname.sh")
The child process can inherit the parent's environment but cannot make any changes. Similarly the parent cannot have access to the child's environment as well. Hence to catch environment of the child in parent the child should print the values as shown in the bellow code. The below code will set already existing environment variables as well, but this can be optimized
# perlscript.pl
my $env_val = `. setnameenv.sh; env`;
my #env_list = split "\n", $env_str;
foreach (#env_list)
{
/([\w_]+)=(.*)/;
$ENV{$1} = $2;
}
print `. getnameenv.sh`;
find the actual explanation in this SO answer
Variables are only exported for the child processes.
You cannot export variables back to the father process.
You'll need another way to transport variables back to the father or the brothers.
For example, here is a example where all exported variables are saved and read from a file :
#!/bin/dash
# setnameenv.sh
export my_msg='hello world!'
export > savedVariables.sh
and
#!/bin/dash
# getnameenv.sh
. ./savedVariables.sh
echo "$my_msg"
Note : this works with dash. bash generates one line he cannot read back.
Assume a script master.sh, which is called as
./foo/bar/master.sh
and contains the lines
#!/bin/zsh
. ./x/y/slave.sh
Is it possible to find out from within slave.sh, that the script which is doing the sourcing, is ./foo/bar/master.sh ?
I can not use $0 here, because this would return ./x/y/slave.sh.
I'm using zsh 5.0.6
one way you can achieve this is that for the child script to take as optional argument the name of the caller. Thus this would be accessible with `$1``
ex:
#!/bin/zsh
# master/leader
. ./x/y/slave.sh $0 # or hardcoded path
#!/bin/zsh
# slave/worker
echo "Here is my master $1"
(you can also do another custom protocol using a environment variable set by the master)
(this solution would also works on bash, and other shell)
The information can already be obtained in zsh right now (thanks to Bart Schaefer, who pointed out to me the existence of the variable functrace in the zsh/parameter module):
#!/bin/zsh
# slave/worker
zmodload zsh/parameter
echo "Here is my master ${functrace[$#functrace]%:*}"
The '%:*' is necessary, because the entries in the functrace array also contain the line number of the call.
If I am writing a bash script, and I choose to use a config file for parameters. Can I still pass in parameters for it via the command line? I guess I'm asking can I do both on the same command?
The watered down code:
#!/bin/bash
source builder.conf
function xmitBuildFile {
for IP in "{SERVER_LIST[#]}"
do
echo $1#$IP
done
}
xmitBuildFile
builder.conf:
SERVER_LIST=( 192.168.2.119 10.20.205.67 )
$bash> ./builder.sh myname
My expected output should be myname#192.168.2.119 and myname#10.20.205.67, but when I do an $ echo $#, I am getting 0, even when I passed in 'myname' on the command line.
Assuming the "config file" is just a piece of shell sourced into the main script (usually containing definitions of some variables), like this:
. /etc/script.conf
of course you can use the positional parameters anywhere (before or after ". /etc/..."):
echo "$#"
test -n "$1" && ...
you can even define them in the script or in the very same config file:
test $# = 0 && set -- a b c
Yes, you can. Furthemore, it depends on your architecture of script. You can overwrite parametrs with values from config and vice versa.
By the way shflags may be pretty useful in writing such script.