Is it possible, using bash, to create a view/virtual file that when opened combines 2 files into 1?
example:
FILE_META_1.txt
FILE_META_2.txt
combines into
FILE_META.txt
In general, this is not possible. I assume you mean you want to logically link 2 files without creating a 3rd file that is the sum of the 2 files. I've often wanted this feature also. It would have to be done at the kernel level or via a special file system, maybe use FUSE. UnionFS provides this for directories, but not for files. FuseFile looks like it does what you want. Also take a look at the Logic File System.
You can open them stream-like wise with process substitution:
cat <(cat FILE_META_1.txt; cat FILE_META_2.txt;)
<(*) here expands to a named pipe path which you could open and access like a file for input.
Related
I am looking for some kind of logic in linux where I can place files with same name in a directory or file system.
For e.g. i create a file abc.txt, so the next time if any process creates abc.txt it should automatically check and make the file named as abc.txt.1 should be created, then next time abc.txt.2 and so on...
Is there a way to achieve this.
Any logic or third party tools are also welcomed.
You ask,
For e.g. i create a file abc.txt, so the next time if any process
creates abc.txt it should automatically check and make the file named
as abc.txt.1 should be created
(emphasis added). To obtain such an effect automatically, for every process, without explicit provision by processes, it would have to be implemented as a feature of the filesystem containing the files. Such filesystems are called versioning filesystems, though typically the details are slightly different from what you describe. Most importantly, however, although such filesystems exist for Linux, none of them are mainstream. To the best of my knowledge, none of the major Linux distributions even offers one as a distribution-supported option.
Although it's a bit dated, see also Linux file versioning?
You might be able to approximate that for many programs via a customized version of the C standard library, but that's not foolproof, and you should not expect it to have universal effect.
It would be an altogether different matter for an individual process to be coded for such behavior. It would need to check for existing files and choose an appropriate name when opening each new file. In doing so, some care needs to be taken to avoid related race conditions, but it can be done. Details would depend on the language in which you are writing.
You can use BASH expression to achieve this. For example if I wanted to make 10 files all with the same name, but having a unique number value I would do the following:
# touch my_file{01..10}.txt
This would create 10 files starting at 01 all the way to 10. This method is also hand for looping over files in a sequence or if your also creating directories.
Now if i am reading you question right your asking that if you move a file or create a file in a directory. you would want the a script to automatically create a new file for you? If that is the case then just use a test and if there is a file move that file and mark it. Me personally I use time stamps to do so.
Logic:
# The [ -f ] tests if the file is present
if [ -f $MY_FILE_NAME ]; then
# If the file is present move the file and give it the PID
# That way the name will always be unique
mv $MY_FILE_NAME $MY_FILE_NAME_$$
mv $MY_NEW_FILE .
else
# Move or make the file here
mv $MY_NEW_FILE .
fi
As you can see the logic is very simple. Hope this helps.
Cheers
I don't know about Your particular use case, but You may try to look at logrotate:
https://wiki.archlinux.org/index.php/Logrotate
I have two directories that I want to compare the contents of. The structure of all the sub-directories underneath the top level directories that I want to compare is the same, but the file names are slightly different. E.g. I want to know if /d1/foo/bar/somefile-v2.txt is different to /d2/foo/bar/somefile-v3.txt.
I've found references that say it should be possible to do this using diff, but I can't find any documentation that tells me how to specify how diff maps from one file name to the other.
What you're referring to is diff's recursive option which in turn will compare all the files in a directory against another.
By reading your comments one can assume that:
File names are different
Using diff alone will not suffice.
You can just write a shell script to take care of this, but to answer your question: No, this can't be done with diff alone.
Note: you can use wildcards for the whole 1 character difference thing but it won't work recursively the way you want it to work.
I want to create some bash scripts. They're actually going to be build scripts for Scala, so I'm going to identify them with my own .bld extension. They will be a sort of sub type of a shell script. Hence I want them to be easily recognised as a shell script. Should I call them
ProjectA.bld.sh //or
ProjectA.sh.bld
Edit: My natural inclination would be to go for the former but .tar.gz files seem to follow the latter naming convention.
A shell script doesn't mind what you call it.
It just needs to be..
executable (chmod +x)
in your path
contain a "shebang" as it's first line #!/bin/sh
The shebang determines which program is used to execute your script.
Call it ProjectA.bld.sh (or preferably buildProjectA.sh).
The .sh extension (although not necessary for the script to run) will allow you and everyone else to easily recognise it as a shell script.
While for the most part, naming conventions like this don't really matter at all to Unix/Linux, the usual convention is for the "extensions" to be in the order of the steps used to create the file. So, for example, a file named foo.tar.bz2.gpg.part01 would indicate a sequence of operations like the following:
Use tar to create foo.tar, which contains some other files
Use bzip2 to compress foo.tar into foo.tar.bz2
Use gnupg to encrypt foo.tar.bz2 into foo.tar.bz2.gpg
Use split or something similar to break the file into chunks for transmission/storage, resulting in one or more foo.tar.bz2.gpg.part* files.
The naming conventions are mostly just for human semantic meaning, though, and there's nothing stopping you from doing exactly the opposite, or even something completely random, except your own ability to remember exactly what you did...
I am trying to write a script or a piece of code to archive files, but I do not want to archive anything that is currently open. I need to find a way to determine what files in a directory are open. I want to use either Perl or a shell script, but can try use other languages if needed. It will be in a Linux environment and I do not have the option to use lsof. I have also had inconsistant results with fuser. Thanks for any help.
I am trying to take log files in a directory and move them to another directory. If the files are open however, I do not want to do anything with them.
You are approaching the problem incorrectly. You wish to keep files from being modified underneath you while you are reading, and cannot do that without operating system support. The best that you can hope for in a multi-user system is to keep your archive metadata consistent.
For example, if you are creating the archive directory, make sure that the number of bytes stored in the archive matches the directory. You can checksum the file contents before and after reading the filesystem and compare that with what you wrote to the archive and perhaps flag it as "inconsistent".
What are you trying to accomplish?
Added in response to comment:
Look at logrotate to steal ideas about how to handle this consistently just have it do the work for you. If you are concerned that rename of files will make processes that are currently writing them will break things, take a look at man 2 rename:
rename() renames a file, moving it
between directories if required. Any
other hard links to the file (as
created using link(2)) are unaffected.
Open file descriptors for oldpath are
also unaffected.
If newpath already exists it will be atomically replaced (subject
to a few conditions; see ERRORS
below), so that there is no point at
which another process attempting to
access newpath will find it missing.
Try ls -l /proc/*/fd/* as root.
msw has answered the question correctly but if you want to file the list of open processes, the lsof command will give it to you.
Under Linux I can open a directory using opendir and then use readdir to get the filenames.
I have been experimenting with scandir and thought "great I can search for the files in this directory that I want by passing in a custom filter", and sort using a custom sort where I want to sort by creation date. But then I realised how limited the dirent structure is. It contains only minimal information.
Is this the only API possible? i.e. do I have to stat every single file to get it's size for sorting? Is this how ls -t works?
That is, indeed, how ls -t works, as 'strace ls -t' will confirm. Historically, a UNIX directory was just a special file containing a list of file names, and applications were expected to read and parse that "file" themselves. Naturally, that led to problems when newer file systems were developed that expanded the fixed length of file names, so the opendir/readdir/closedir interface was developed to abstract away the filesystem directory implementation. But the limitation on what is directly available in a directory listing remains.
POSIX does not have any facility for storing creation time, much less retrieving it.