I've always believed that environment variables live within the shell current user is logged into. However recently I've begun working on a shell of my own and learning more about how Linux works under the hood. Now it seems to me, that the environment is shell-independent and handled elsewhere (in the kernel?). So my question is how exactly does it work? Which part of the system is responsible for holding the environment?
Also for instance Bash makes the distinction between export-ed and unexported variables, the latter of which are not inherited by a subshell. Does that mean that each process is the system has it's own set of shell variables?
Yes each process will have its own set of enviornment.
You can find them at
cat /proc/<pid>/environ
Related
Reading and writing environment variables in Node.js is done using the process.env object.
For instance:
process.env.foo evaluates to the env var foo
process.env.bar = 'blah' sets the value of the env var bar to blah
delete process.env.baz deletes the environment variable baz
From trial and error, and the lack of a callback, I assume that these actions are synchronous, but I found no reference to it process.env documentation.
Is env var access synchronous or asynchronous in Node.js?
Addendum: Why I believe this question to be non-trivial
Following the comments: Reading and writing the environment variables might mean that the process needs to communicate with the operating system, or perform some sort of blocking I/O operations.
Therefore, it makes sense to ask whether the environment variables are stored as a local object in memory without any synchronization, or otherwise sent to the operating system in a blocking manner.
Moreover, implementation may vary between operating systems and the official documentation lacks any promise to a non-blocking operation.
I think the "synchronous"/"asynchronous" may be a bit misleading.
I guess the actual question is: Is reading from or writing to process.env expensive? Does it perform a blocking operation with the operating system?
The short answer is Yes, it can be expensive.
For more background info and how much it can impact some apps, see this GitHub issue. There it was already stated that the documentation should be updated to make it clear that accessing process.env is slow in 2015 but it hasn't happened yet.
You can actually see the implementation for process.env in the node.js source code where it's obvious that any access will call one of the functions defined from here onwards.
Note: At the time of writing, this was defined in node.cc in a more straight-forward way. The links above still point to the old implementation. Newer versions of node have process.env implemented in a separate file node_env_var.cc which can be found here, but it has more encapsulation, making it harder to follow for the purpose of this explanation.
Depending on the platform, this may have more or less of an impact.
It becomes most obvious on Windows, because there you can view a process' current environment from the outside (while in Linux, the /proc/.../environ file will retain its original contents when the environment was changed with setenv).
For example:
node -e "process.env.TEST = '123'; setInterval(() => {}, 1000);";
This will start a node process which creates a TEST environment variable in the current process' environment and then wait forever.
Now we can open a tool like Process Explorer or Process Hacker and look at the environment of the node process:
And lo and behold, the variable is there. This proves in another way that writing to process.env does in fact access the operating system.
Also, because the object actually queries all data from the OS, it means that it even behaves different than a normal object. Again, Windows example (because it's most quirky):
Windows matches environment variables case-insensitive.
> process.env.TEST = '123'
'123'
> process.env.tEsT
'123'
Windows has hidden environment variables starting with = which cannot be changed through normal means and which are not enumerated. node.js replicates these semantics. The =X: variables in particular represent the current directory in specific drives (yes, Windows stores them per drive).
> Object.keys(process.env).filter(k => k === '=Z:')
[]
> process.env['=Z:']
'Z:\\'
> process.env['=Z:'] = 'Z:\\Temp'
'Z:\\Temp'
> process.env['=Z:']
'Z:\\'
> process.chdir('Z:\\Temp')
undefined
> process.env['=Z:']
'Z:\\Temp'
Now, somebody might think (similar to what was proposed in the GitHub issue that I linked) that node.js should just cache process.env in an actual object, and for child process creation read the environment from the cached object. This is not advisible for the following reasons:
They would need to copy the semantics of the underlying platform and reimplement them. As you can see in the above example for Windows, this would at some point end up in intercepting chdir and trying to automatically update the relevant =X: variable of the affected drive (and then it wouldn't work if a native plugin would change the current directory), or access the OS only for some variables, and therein lies madness and huge potentional for obscure bugs.
This would break applications which read a process' environment from the outside (like Process Explorer), as they would see incorrect values.
This would create inconsistencies if a native module would access the environment variables in its own from C++ code, because they would now have a different state than the cached object.
This would cause childprocesses to not inherit the correct variables if the child process were started by a native module (for the same reason as above).
This should also explain why it is a bad idea to do process.env = JSON.parse(JSON.stringify(process.env)) in your code. For one, it would break case-insensitivity on Windows (and you can't possibly know what modules which some other module requires may depend on that), and apart from that it would of course cause tons of other problems as described above.
Actually it is an normal object to make you could get the environment variables of current process, after all them are just some variables for carry some setting to a program. Nodejs just set a normal object for them after nodejs program read them. Although documentation not write them but it write this is an object and following things:
It is possible to modify this object, but such modifications will not
be reflected outside the Node.js process. In other words, the
following example would not work:
$ node -e 'process.env.foo = "bar"' && echo $foo
While the following will:
process.env.foo = 'bar';
console.log(process.env.foo);
Assigning a property on process.env will implicitly convert the value to a string.
This is enough to explain your problem.
Nearly all linux courses say that init process, given the run level, will execute appropriate shell scripts to initialize the enivronment. But non of the courses describe in detail how init process does it.
As I understand the init process is basically a C program, much like any Hello World C code. Only much more sophisticated. Does anyone knows how this C program actually runs through all the scripts and invokes them?
I would really appreciate any answer and especially if you have a link to an example source code.
You can find explanations of what it does in different documentation:
http://www.centos.org/docs/5/html/5.1/Installation_Guide/s2-boot-init-shutdown-init.html
http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=4
and you can find its source code over there:
init.h
init.c
basically, init as process 1, has for role to fork() every application on your system. If you boot linux with the command line init=/bin/sh at boot time, process 1 forked by the kernel will be a shell. The sysvinit program makes it a bit more easy to handle a complex boot. It adds the concept of runlevels, define basic environment etc.. so that makes it easy to boot a system and have many services, and not only a shell. All that part is well explained in the documentations I gave you.
Does anyone knows how this C program actually runs through all the scripts and invokes them?
Well, is as simple as in your question. When you boot your system, init reads the inittab file, figures out what are your preferences (what is the default runlevel? what program to spawn? how many consoles?..), and for the chosen runlevel will fork a shell that will execute the startup script. Then that shell script will makes its way up to the shell script you activated from /etc/init.d. Usually the shell script part is very distribution-specific, that's why I gave you two links about that, and you may find it is different on ubuntu and debian...
For more details on the source code, you may want to start at the bottom of init.c which contains init's mainloop.
And +1 on your question for your curiosity!
I have a script that exports an environment variable and starts some subscripts.
export LOGLEVEL="1"
/home/myuser/bin/myscript1.sh
/home/myuser/bin/myscript2.sh
LOGLEVEL is available to the processes started from the subscripts. How can I change the environment variable LOGLEVEL?
I have tried to set the variable with export LOGLEVEL="5" but that`s not working.
In general, you can only influence a process's environment variables at the time the process starts up. If you need to communicate a change to a running process, the environment isn't the right tool.
However, this question has some answers that suggest ways to overcome this limitation.
Edited to add in light of discussion in the question's comments: A fairly good way of communicating occasionally changing setup to a running process is to designate a configuration file where the LOGLEVEL value is set, send a SIGHUP to the process, and have the process reread the configuration file upon receipt of SIGHUP.
I use Terminal.app and iTerm, both of which support running multiple shells simultaneously via multiple tabs and multiple windows. I often use this feature, but as a result, if I want to change an environment variable setting, I generally have to run the same command once in every single tab and window I have open -- as well as any new tabs or windows that I open in the future. Is it possible for my shells to communicate with each other such that I can change an environment variable once, and have that change propagate to all my other currently running shells?
I know that I can statically set env variables in a startup file like .bashrc. I also know that I can have subshells inherit the environment of parent shells, either normally or via screen. Neither of those options address this question. This question is specifically about dynamically changing the environment of multiple currently-running shells simultaneously.
Ideally, I'd like to accomplish this without writing the contents of these variables to disk at any point. One of the reasons I want to be able to do this is so that I can set sensitive information in an env variable, such as hashed passwords, and refer to them later on in other shells. I would like to be able to set these variables once when I log in, and be able to refer to them in all my shells until I log out, or until the machine is restarted. (This is similar to how ssh-agent works, but as far as I'm aware, ssh-agent will only store SSH keys, not env variables.)
Is it possible to make shells communicate like this?
Right. Since each process has it's own copy of the environment variables, you can't magically change them all at once. If you bend your mind enough though, there are strange workarounds.
For instance, if you currently have a command you run to update each one, you can automate running that command. Check the bash man page for PROMPT_COMMAND, which can run a command each time the bash prompt is printed. Most shells have something similar.
As far as not putting a hashed password on disk because you are pulling it from an envvar instead of something like ssh-agent...that would be a whole 'nother topic.
Unless you write your own shell, you can't. ssh-agent works by having each SSH client contact it for the keys, but most common shells have no similar mechanism.
We have an stub that we are launching from inittab that execv's our process. (ARM Linux Kernel 2.6.25)
When testing the process it fails only if launched from inittab and execv'd. If launched on the command line it work perfectly, every time.
The process makes heavy use of SYS V IPC.
Are there any differences between the two launch methods that I should be aware of?
As Matthew mentioned it, it is probably an env variable issue. Try to dump yout env list before calling your program in both case - through the stub or 'by hand'.
BTW It could help a lot if you could provide more information why your program did crash. Log file ? core dump/gdb ? return value from execve ?
Edit:
Other checks: are you sure to pass exactly the same parameter list (if there are parameters)?
To answer your question , there is no differences between the 2 methods. Actually your shell fork() and finally call execve() to launch your process, feeding it with parameters you've provided by hand, and the environement variables you've set in your shell. Btw when launching your program through init it could launch it during an early stage of your machine startup. Are you sure everything is ready for the good running of your application at that point?
Could it be an issue of environment variables? If so, consider using execve or execle with an appropriate envp argument.
The environment variable suggestion is pretty good - specifically I'd check $PATH to make sure your dependent libraries are being found (if you have any). Another thing you could check is, are you running under the same uid/gid when run as inittab?
And if you replace your stub with a shell script ?
If it works from the command line, it should work from a shell script, and you can know wether it is your stub or the fact that it is in inittab.
Could it be a controlling tty issue ?
Another advantage of the shell script is you can edit it and strace your program to see where it fails
Was a mismatched kernel/library issue. Everything cleaned up after a full recompile.