I'm writing a program that needs to be able to read in the time zone files on Linux. And that means that I need to be able to consistently find them across distros. As far as I know, they are always located in /usr/share/zoneinfo. The question is, are they in fact always located in /usr/share/zoneinfo? Or are there distros which put them elsewhere? And if so, where do they put them?
A quote from tzset(3):
The system timezone directory used depends on the (g)libc version.
Libc4 and libc5 use /usr/lib/zoneinfo,
and, since libc-5.4.6,
when this doesn't work, will try /usr/share/zoneinfo.
Glibc2 will use the environment
variable TZDIR, when that exists. Its
default depends on how it was installed, but normally is
/usr/share/zoneinfo.
Note, however, that nothing prevents some perverse distro from patching libc and placing the files wherever they want.
The public-domain time zone database contains the code and data to handle time zones on Linux.
The public-domain time zone database
contains code and data that represent
the history of local time for many
representative locations around the
globe. It is updated periodically to
reflect changes made by political
bodies to time zone boundaries, UTC
offsets, and daylight-saving rules.
This database (often called tz or
zoneinfo) is used by several
implementations, including the GNU C
Library used in GNU/Linux, FreeBSD,
NetBSD, OpenBSD, Cygwin, DJGPP, AIX,
Mac OS X, OpenVMS, Oracle Database,
Solaris, Tru64, and UnixWare.
That covers a lot of system but I can only agree with Roman that nobody can be prevented from creating a distribution that differs for whatever reasons. The existence and location of a zonezinfo file is not covered by any official standard as far as I know. The standards (e.g. POSIX and XPG4) only establish the API.
Related
I recently learned that different OSes and even different filesystems under the same OS support different subsets of the timestamps returned by lstat.
The Stats object returned gives us four times, each in two different flavours.
js Date objects:
atime: the last time this file was accessed expressed in milliseconds since the POSIX Epoch
mtime: the last time this file was modified ...
ctime: the last time the file status was changed ...
birthtime: the creation time of this file
(atimeMs, mtimeMs, ctimeMs, and birthtimeMs are js Date object versions of each of the above)
"Modified" means the file's contents were changed by being written to etc.
"Changed" means the file's metadata such as owners and permissions was changed.
Linux has traditionally never supported the concept of birth time, but as more newer filesystems did support it, it has recently had support added to hopefully all relevant layers of the Linux stack if I have read correctly.
But Windows and Mac both do support birth time as do their native filesystems.
Windows on the other hand did not traditionally support a concept of file change separate from file modification. But to comply to POSIX it added support at the API level and to NTFS. (It doesn't seem to be exposed anywhere in the GUI or commandline though). FAT fs does not support it.
When I call lstat on a file on Windows on an NTFS drive the results for ctime look good. When I call it on a file on a FAT drive, ctime contains junk. (In my case it's always 2076-11-29T08:54:34.955Z for every file.)
I don't know if this is a bug.
I don't know what birthtime returns on Linux on filesystems that don't support it. Hopefully null or undefined but perhaps also garbage. I also don't know what Linux or Mac return in ctime for files on FAT volumes.
So is there a way in Node to get info on which of these features are supported for a given file/path/fs?
The Wikipedia page on Core dump says
In Unix-like systems, core dumps generally use the standard executable
image-format:
a.out in older versions of Unix,
ELF in modern Linux, System V, Solaris, and BSD systems,
Mach-O in OS X, etc.
Does this mean a core dump is executable by itself? If not, why not?
Edit: Since #WumpusQ.Wumbley mentions a coredump_filter in a comment, perhaps the above question should be: can a core dump be produced such that it is executable by itself?
In older unix variants it was the default to include the text as well as data in the core dump but it was also given in the a.out format and not ELF. Today's default behavior (in Linux for sure, not 100% sure about BSD variants, Solaris etc.) is to have the core dump in ELF format without the text sections but that behavior can be changed.
However, a core dump cannot be executed directly in any case without some help. The reason for that is that there are two things missing from a simple core file. One is the entry point, the other is code to restore the CPU state to the state at or just before the dump occurred (by default also the text sections are missing).
In AIX there used to be a utility called undump but I have no idea what happened to it. It doesn't exist in any standard Linux distribution I know of. As mentioned above (#WumpusQ) there's also an attempt at a similar project for Linux mentioned in above comments, however this project is not complete and doesn't restore the CPU state to the original state. It is, however, still good enough in some specific debugging cases.
It is also worth mentioning that there exist other ELF formatted files that cannot be executes as well which are not core files. Such as object files (compiler output) and .so (shared object) files. Those require a linking stage before being run to resolve external addresses.
I emailed this question the creator of the undump utility for his expertise, and got the following reply:
As mentioned in some of the answers there, it is possible to include
the code sections by setting the coredump_filter, but it's not the
default for Linux (and I'm not entirely sure about BSD variants and
Solaris). If the various code sections are saved in the original
core-dump, there is really nothing missing in order to create the new
executable. It does, however, require some changes in the original
core file (such as including an entry point and pointing that entry
point to code that will restore CPU registers). If the core file is
modified in this way it will become an executable and you'll be able
to run it. Unfortunately, though, some of the states are not going to
be saved so the new executable will not be able to run directly. Open
files, sockets, pips, etc are not going to be open and may even point
to other FDs (which could cause all sorts of weird things). However,
it will most probably be enough for most debugging tasks such running
small functions from gdb (so that you don't get a "not running an
executable" stuff).
As other guys said, I don't think you can execute a core dump file without the original binary.
In case you're interested to debug the binary (and it has debugging symbols included, in other words it is not stripped) then you can run gdb binary core.
Inside gdb you can use bt command (backtrace) to get the stack trace when the application crashed.
I have a question on Zen of Unix / GNU Linux, which looks like an aha-moment for me...
Apparently ANY standard Unix/Linux program has a config file, which the most are located at /etc directory.
Can we derive the concept as follow:
1- As an application developer you should design your software which have a customiztion file (possibly located at /etc.)
2- Then, Admins or users can SET these configs based on their needs and run your program.
3- Changing the behavior of your program should ONLY depends on its config file.
If you're asking whether this is true, that tends to be the convention, yes. Keep in mind that developers are free to design their programs to run however they want to and that they tend to follow this pattern only for convenience of similarity.
Other patterns you may see:
Programs with no global settings and only per user settings may store their settings in ~/.[something], or maybe somewhere else entirely. Many programs do this AND use /etc. Bash is a good example, using /etc/profile/.bashrc for default settings and ~/.bashrc for user settings.
Very large standalone installations of some programs may package all of their files into their own .../etc, .../bin, etc.. directories, and will not use the typical system directories at all. An example of this is charybdis, an ircd, which stores everything in a folder specified at compile time (Mine lives in /var/ircd, so I have /var/ircd/etc, /var/ircd/bin, /var/ircd/lib, ...)
OSX is a certified Unix and tries not to use etc - in effect only Apple supplied programs should change /etc, they supply alternatives.
However for all OSs including Windows you do have a separate configuration/customisation file (or in Windows the registry) and there probably need to be two of these. One that is set and controlled by admins and one for changes the user makes. The former of these can use /etc for Linux see the Filesystem_Hierarchy_Standard
I'm currently developing an application which needs a lot of system and process information, some of which is only available through /proc, and I have some general questions about accessing the structures.
The application will be run on Linux (kernel >= 2.6), not on any other Unix-flavored OS. It should have access to any data in /proc, I can't say what is necessary now as the specifications are not clear yet, but the whole /proc directory is relevant to the application.
First of all: Is there a good documentation which covers all the features added / removed from kernel version to kernel version? One thing I'm curious about in particular is the format of the individual files. Can I take that for granted? Does it change among kernel versions?
Hooking up the parsing process based on the kernel wouldn't be a problem at all, it's just that I couldn't find any good docs on what has changed from version to version which could help me catching parsing errors in beforehand.
In addition: Is there a definite list of features that can be activated / deactivated by kernel options (except of course the /proc-feature itself)? I'm looking for a list of files / directories that only exist with the appropriate options being set in the kernel.
As an example of what I'm thinking of, this is a link to the proc manpage (http://linux.die.net/man/5/proc) which includes a lot of good information, e.g. some options include the earliest kernel version they were available at, some include whether a module is necessary to be loaded. This does not describe the output format of all information though, which is something I need if I want to parse it (e.g. if it is consistent throughout all kernel versions or changed at some point).
The second thing I'm wondering about is what happens if the process queried dies while being queried. What is my time interval? For example if I'm going to fetch a list of processes reading all the structures, and parse them one after another, what happens if my process x dies before I get to read it? Even if I check if the directory exists, it could still be gone one application call later.
Last but not least: Is there any major distribution out there that is not mounting proc?
From what I understand, a lot of common tools are based on the /proc interface such as lsmod or free, so I'm guessing that I can expect /proc to exist almost always.
The /proc interfaces are pretty stable (unlike the /sys interfaces), even if nothing is guaranteed. Almost all changes are backwards compatible, at least if they've been around for a few versions. You should
stick to the documented interfaces to be safe. If a file exists, its format may be extended in later versions, but normally in a backwards compatible way, e.g. adding columns to a table. The parts that are most at risk of disappearing are parts concerning hardware susbystems such as ACPI or SCSI, which are migrating to /sys (with a long transition period when both exist).
Most of the information is architecture-independent, except for hardware information (e.g. /proc/cpuinfo has very different fields on different architectures).
The main documentation is Documentation/filesystems/proc.txt in the kernel source. Consider proc(5) to be the overview and proc.txt to be the fine details. The kernel documentation is often incomplete, so don't be surprised if you need to resort to reading the source sometimes.
Most optional parts of /proc are activated by default if the driver whose data it exposes is included in the kernel. The exceptions are mostly related to hardware features that rarely need to be accessed from outside the kernel; if you need to access these features, you're probably already expecting to need to dig deeper. Look through Kconfig files in the kernel source for detailed information.
Process data (or hardware data related to removable hardware or provided by unloadable modules) can disappear under your nose. Most files under /proc can be read atomically, with a single read call with a reasonably-sized buffer; if you perform multiple read calls in sequence, drivers are supposed to guarantee that you get well-formed data. There is no way to guarantee atomicity between reads of separate files; if you're reading information about a process, this process can die at any time, and in principle could even be replaced by another process with the same PID before you're finished.
As it says in the description of /proc, “everyone should say Y here”. All desktop/server Linux systems and most embedded Linux systems must have /proc; a lot of things, including ps and other process management commands, many filesystem and device-related tools, and module loading, require it. The only systems that might be able to dispense with /proc are very small single-purpose embedded systems that support a single hardware configuration and run a fixed set of programs. You can count on its being here.
I have an application that runs on an embedded Linux device and every now and then changes are made to the software and occasionally also to the root file system or even the installed kernel.
In the current update system the contents of the old application directory are simply deleted and the new files are copied over it. When changes to the root file system have been made the new files are delivered as part of the update and simply copied over the old ones.
Now, there are several problems with the current approach and I am looking for ways to improve the situation:
The root file system of the target that is used to create file system images is not versioned (I don't think we even have the original rootfs).
The rootfs files that go into the update are manually selected (instead of a diff)
The update continually grows and that becomes a pita. There is now a split between update/upgrade where the upgrade contains larger rootfs changes.
I have the impression that the consistency checks in an update are rather fragile if at all implemented.
Requirements are:
The application update package should not be too large and it must also be able to change the root file system in the case modifications have been made.
An upgrade can be much larger and only contains the stuff that goes into the root file system (like new libraries, kernel, etc.). An update can require an upgrade to have been installed.
Could the upgrade contain the whole root file system and simply do a dd on the flash drive of the target?
Creating the update/upgrade packages should be as automatic as possible.
I absolutely need some way to do versioning of the root file system. This has to be done in a way, that I can compute some sort of diff from it which can be used to update the rootfs of the target device.
I already looked into Subversion since we use that for our source code but that is inappropriate for the Linux root file system (file permissions, special files, etc.).
I've now created some shell scripts that can give me something similar to an svn diff but I would really like to know if there already exists a working and tested solution for this.
Using such diff's I guess an Upgrade would then simply become a package that contains incremental updates based on a known root file system state.
What are your thoughts and ideas on this? How would you implement such a system? I prefer a simple solution that can be implemented in not too much time.
I believe you are looking wrong at the problem - any update which is non atomic (e.g. dd a file system image, replace files in a directory) is broken by design - if the power goes off in the middle of an update the system is a brick and for embedded system, power can go off in the middle of an upgrade.
I have written a white paper on how to correctly do upgrade/update on embedded Linux systems [1]. It was presented at OLS. You can find the paper here: https://www.kernel.org/doc/ols/2005/ols2005v1-pages-21-36.pdf
[1] Ben-Yossef, Gilad. "Building Murphy-compatible embedded Linux systems." Linux Symposium. 2005.
I absolutely agree that an update must be atomic - I have started recently a Open Source project with the goal to provide a safe and flexible way for software management, with both local and remote update. I know my answer comes very late, but it could maybe help you on next projects.
You can find sources for "swupdate" (the name of the project) at github.com/sbabic/swupdate.
Stefano
Currently, there are quite a few Open Source embedded Linux update tools growing, with different focus each.
Another one that is worth being mentioned is RAUC, which focuses on handling safe and atomic installations of signed update bundles on your target while being really flexible in the way you adapt it to your application and environment. The sources are on GitHub: https://github.com/rauc/rauc
In general, a good overview and comparison of current update solutions you might find on the Yocto Project Wiki page about system updates:
https://wiki.yoctoproject.org/wiki/System_Update
Atomicity is critical for embedded devices, one of the reasons highlighted is power loss; but there could be others like hardware/network issues.
Atomicity is perhaps a bit misunderstood; this is a definition I use in the context of updaters:
An update is always either completed fully, or not at all
No software component besides the updater ever sees a half installed update
Full image update with a dual A/B partition layout is the simplest and most proven way to achieve this.
For Embedded Linux there are several software components that you might want to update and different designs to choose from; there is a newer paper on this available here: https://mender.io/resources/Software%20Updates.pdf
File moved to: https://mender.io/resources/guides-and-whitepapers/_resources/Software%2520Updates.pdf
If you are working with the Yocto Project you might be interested in Mender.io - the open source project I am working on. It consists of a client and server and the goal is to make it much faster and easier to integrate an updater into an existing environment; without needing to redesign too much or spend time on custom/homegrown coding. It also will allow you to manage updates centrally with the server.
You can journal an update and divide your update flash into two slots. Power failure always returns you to the currently executing slot. The last step is to modify the journal value. Non atomic and no way to make it brick. Even it if fails at the moment of writing the journal flags. There is no such thing as an atomic update. Ever. Never seen it in my life. Iphone, adroid, my network switch -- none of them are atomic. If you don't have enough room to do that kind of design, then fix the design.