Suppose I have a folder and I want only one instance of my application working on it at a time. I can only synchronize via the filesystem itself. Often times this is a accomplished with something like a .lock_file where if that's present I know another instance is currently using it. Are there any standard libraries that handle this sort of thing?
If you are using C/C++, see fclnt or flock :
Locking files in linux with c/c++
If you are using java, see FileChannel lock method
and
How can I lock a file using java (if possible)
You also can check for the existence of the .lock_file opening it with
open(pathname, O_CREAT | O_EXCL, 0644),
see open man page, it creates and opens the file and returns EEXIST if pathname exists.
In java, calling to File method createNewFile() can be use to create atomically the .lock_file
Related
We are using System.gc method to halt JVM to delete temp directory,because previously it was not getting deleted,but i got to know that using System.gc or Runtime.getRuntime() is a bad practice,so I Im thinking to use methods others than this ,kindly suggest which one to use.
Manage your resources manually instead of relying on finalizers. I.e. delete the temp directory at the end of the method that created it or if you're using asynchronous control flows then add a CompletableFuture to the end that does the removal.
Use try-with-resources statements to ensure that files only remain open within a limited scope.
On posix systems you can simply delete (unlink) files and (rmdir) directories even if they are still referenced by some file descriptors.
I have a rather peculiar file format to work with:
Every line begins with the checksum of its content, followed by a new-line-character.
It looks like this:
[CHECKSUM OF LINE_1][LINE_1]\n
[CHECKSUM OF LINE_2][LINE_2]\n
[CHECKSUM OF LINE_3][LINE_3]\n
...
My goal: To allow any application to work with these files like they would work with any other text file - unaware of the additional checksums at the beginning of each line.
Since I work on a linux machine with debian wheezy (kernel 3.18.26) I want to use the LD_PRELOAD-mechanism to override the relevant file functions.
I have seen something like this with zlibc on https://zlibc.linux.lu/index.html - with an explanation of how it works ( https://zlibc.linux.lu/zlibc.html#SEC8 ).
But I dont get it. They only replace the file-opening functions. No read. No write. no fseek. Nothing. So how does it work?
Or - which functions would I have to intercept to handle every read or write operation on this file and handle them accordingly?
I didn't exactly check how it works but the reason seems to be quite simple.
Possible implementation:
zlibc open:
uncompress file you wanted to open to some temporary file
open this temporary file instead of yours
zlibc close:
Compress temporary file
Override original file
In this case you don't need to override read/write/etc because you can use original ones.
In your case you have two possible solutions:
open, that make a copy of your file with striped checksums. close that calculates checksums and override original file
read and write that are able to skip/calculate checksums.
Ad 2.
From What is the difference between read() and fread()?:
fread() is part of the C library, and provides buffered reads. It is
usually implemented by calling read() in order to fill its buffer
In this case I believe that overriding open and close will be less error prone because you can safely reuse original read, write, fread, fseek etc.
Is it possible to open a file knowing its inode?
ls -i /tmp/test/test.txt
529965 /tmp/test/test.txt
I can provide path, inode (above 529965) and I am looking to get in return a file descriptor.
This is not possible because it would open a loophole in the access control rules. Whether you can open a file depends not only on its own access permission bits, but on the permission bits of every containing directory. (For instance, in your example, if test.txt were mode 644 but the containing directory test were mode 700, then only root and the owner of test could open test.txt.) Inode numbers only identify the file, not the containing directories (it's possible for a file to be in more than one directory; read up on "hard links") so the kernel cannot perform a complete set of access control checks with only an inode number.
(Some Unix implementations have offered nonstandard root-only APIs to open a file by inode number, bypassing some of the access-control rules, but if current Linux has such an API, I don't know about it.)
Not exactly what you are asking, but (as hinted by zwol) both Linux and NetBSD/FreeBSD provide the ability to open files using previously created “handles”: These are inode-like persistent names that identify a file on a file system.
On *BSD (getfh and fhopen) using this is as simple as:
#include <sys/param.h>
#include <sys/mount.h>
fhandle_t file_handle;
getfh("<file_path>", &file_handle); // Or `getfhat` for the *at-style API
// … possibly save handle as bytes somewhere and recreate it some time later …
int fd = fhopen(&file_handle, O_RDWR);
The last call requiring the caller to be root however.
The Linux name_to_handle_at and open_by_handle_at system calls are similar, but a lot more explicit and require the caller to keep track of the relevant file system mount IDs/UUIDs themselves, so I'll humbly link to the detailed example in the manpage instead. Beware, that the example is not complete if you are looking to persist the handles across reboots; one has to convert the received mount ID to a persistent filesystem identifier, such as a filesystem UUID, and convert that back to a mount ID later on. In essence they do the same however. And just like on *BSD using the later system call requires elevated privileges (CAP_DAC_READ_SEARCH to be exact).
If I've got a handle to an open file, is it possible to create a hard link to that file after all references to it have been removed from the filesystem?
For example, something like this:
fd = fopen("/tmp/foo", "w");
unlink("/tmp/foo");
fwrite(fd, "Hello, world!\n");
create_link_from_fd(fd, "/tmp/hello");
fclose(fd);
Specifically, I'd like to do this so that I can safely write to large data files, then move them into place atomically without having to worry about cleaning up after myself if my program is killed in the middle of writing the file.
The newly released linux 3.11 offers a solution to this problem with the new O_TMPFILE open(2) flag. With this flag you can create an "invisible" file (i.e. an inode with no hardlinks) in some file system (specified by a directory in that file system). Then, after the file is fully set up, you can create a hardlink using linkat. It works like this:
fd = open("/tmp", O_TMPFILE | O_RDWR, 0600);
// write something to the file here
// fchown()/fchmod() it
linkat(fd, "", AT_FDCWD, "/tmp/test", AT_EMPTY_PATH);
Note that aside from the >=3.11 kernel requirement, this also requires support from the underlying file system (I tried the above snippet on ext3 and it worked, but it did not seem to work on btrfs).
Not generally, no. [Edit: since Linux 3.11 there is now linkat; see safsaf32's answer. This does not work on POSIX systems in general since POSIX linkat is restricted to directories only.] There are security considerations here: someone can pass to you an open file descriptor that you could not normally open on your own, e.g.:
mkdir lock; chmod 700 lock
echo secret contents > lock/in
sudoish cmd < lock/in
Here cmd runs as a user who has no permission to open the input file (lock/in) by name, but can still read from it. If cmd could create a new name on the same file system, it could pass the file contents on to a later process. (Obviously it can copy those contents, so this issue is more of a "pass the contents on by mistake" thing than "pass the contents on, on purpose".)
That said, people have come up with ways of "relinking" files by inode/vnode internally (it's pretty easy to do inside most file systems), so you could make your own private system call for it. The descriptor must refer to a real file on the appropriate mount point, of course—there's no way of "relinking" a pipe or socket or device into becoming a regular file.
Otherwise you're stuck with "catch signals and clean up and hope for the best", or a similar trick, "fork off a subprocess, run it, and if it succeeds/fails, take appropriate move/clean-up action".
Edit to add historical note: the above lock example is not particularly good, but back in the days of V6 Unix, MDQS used a fancier version of this trick. Bits and pieces of MDQS survive in various forms today.
On Linux, you might try the unportable trick of using /proc/self/fd by trying to call
char pbuf[64];
snprintf (pbuf, sizeof(pbuf), "/proc/self/fd/%d", fd);
link(pbuf, "/tmp/hello");
I would be surprised if that trick worked after an unlink("/tmp/foo") ... I did not try that.
A more portable (but less robust) way would be to generate a "unique temporary path" perhaps like
int p = (int) getpid();
int t = (int) time(0);
int r = (int) random();
sprintf(pbuf, sizeof(pbuf), "/tmp/out-p%d-r%d-t%d.tmp", p, r, t);
int fd = open (pbuf, O_CREAT|O_WRONLY);
Once the file has been written and closed, you rename(2) it to some more sensible path. You could use atexit in your program to do the renaming (or the removing).
And have some cron job to clean the [old] /tmp/out*.tmp every hour...
hi all:
recently i'm working on make checkpoint on linux process and encountered a problem,it looks like that when i munmap memory map of the executable to current process,the symlink /proc/self/exe is dead.what i want is to make this symlink pointing to a other executable(the one for my resumed processs),is that possible?i tried delete it and recreate, permission denied. english is not my native language, i hope i've made my point,thanx
prctl(PR_SET_MM_EXE_FILE, ...)
Supersede the /proc/pid/exe symbolic link with a new one pointing to a new executable file identified by the file descriptor provided in arg3 argument. The file descriptor should be obtained with a regular open(2) call.
No. /proc is completely managed by the kernel and does not allow changes like that.
But you may be able to start a new process (with fork() perhaps) and map your memory snapshot into that.