With more and more programs installed on my computer, I am tired of seeing lots of dotfiles while I have to access them often. For some reason I won't hide dotfiles when browsing files. Is there a way to move them to a better place I want them to stay (e.g. ~/.config/$PROGCONF) without affecting programs while running?
Symlinks still leave file symbols, which is far from my expectation. I expect that operations like listdirs() won't show the files while opening them uses a redirection.
"For some reason it won't hide dotfiles when browsing files.":
That depends on the file manager you use. nautilus hides it by default and most file managers have an option to "show/hide hidden files". The ls command by default omits out hidden files (files starting with a dot). It lists all files with the option -a.
"Is there a way to move them to a better place":
Programs which have support for "XDG user directories" can store their config files in `~/.config/$PROGRAM_NAME/. If the program doesn't support that and expects the config file to be present in the home directory, there is little you can do (Maybe you can give us a list of what programs' config files you want to move). The process differs for each program.
Let me give an example with vim. Its config file is ~/.vimrc. Lets say you move the file to ~/.config/vim/.vimrc. You can make vim read the file by launching vim using the following command.
vim -u ~/.config/vim/.vimrc
You can modify the .desktop entry or create a new shell script to launch vim using the above command and put it inside /usr/local/bin/ or create shell functions / aliases. You can read more about changing vim's config file location in this SO question.
This arch wiki article has application specific information.
"without affecting programs while running":
It depends on a few factors namely the file system used, the program we are dealing with and so on.
Generally, deleting / moving files only unlinks the file name from an inode and programs read / write files using inodes. Read more here. And most programs read the config file at the start, load the values into memory. They rarely read the config files again. So, if you move your config file while the program is running (assuming the program supports config in both places), you won't see a difference until the program is restarted.
"I expect that operations like listdirs() won't show the files"
I am assuming you are talking about os.listdir() in python. If files are present, os.listdir() will list them, there is little you can change about that. But you can write custom functions to omit out the hidden files from being listed.
This SO question can help with that.
Related
Why is bash(.sh) script not executable by default.
I agree that while we touch any file in linux it is created for reading purpose.
But since file name extensions such as sh and csh are for execution purpose.
Won't it be ideal to touch them in an executable mode.
Question might sound redundant to, but i still thought of asking it :)
Ultimately the answer to this question is that that's not what touch was designed to do.
In fact, touch was not even designed to create files; its primary purpose in life is to change file timestamps. It is merely a side effect of that purpose, and the fact that the designers decided to be a little more generous and create the target file if it doesn't already exist (and if you didn't provide the -c option), that it allows you to create files at all.
It should also be mentioned that there are other techniques that create files, such as redirection (echo 'echo '\''I am a script.'\'';' >|script.sh;). The act of file creation is a generic one, and the whole concept of a file is a generic one. A file is just a byte stream; what goes in the byte stream is unspecified at the file abstraction layer. As #AdamGent mentioned, Windows requires certain types of executable files to have certain extensions in order to be executed properly, but even in Windows, you can put executable code in non-executable-extensioned files, and you can put non-executable content in executable-extensioned files. There's no enforcement of file name / file content correspondence at the file layer.
All of that being said, it would often be a convenience if you could easily create a script in Unix that automatically has executable permission set. I've actually written a script to allow me to edit a new file in vim, and then have its permissions set to executable after write-quitting. The reason this potential convenience has not been standardized into a utility is likely related to the concern about security; you don't want people to accidentally make files executable, because that raises the risk of security holes.
You can always write your own program to create a file and make it executable, perhaps based on the extension of the file name.
Another thing that can be added here is that even shell scripts don't always need to be executable. For example, if you write a script that is only intended to be sourced from existing shell processes (via the source or classic . builtins), then the script does not need to be executable at all. Thus, there are cases where the file extension itself does not provide enough information to determine what the appropriate permissions are for the file.
There is nothing in the file name that says a file is even a script. Common practice perhaps says that .sh and .csh are scripts but there is no such rule.
What makes a file an executable script is the magic number in the beginning of the file. The magic number #! (the shebang, which has many other names as well) means the file is a script. For example
#!/bin/bash
When you try to execute the file (it must then also be set to executable in the permissions), the Linux kernel looks at the first two bytes of the file and finds the magic number (#!). It then knows it is supposed to execute whatever comes after the Shebang with the name of the file as argument followed by the rest of the arguments passed.
If you type the following in the shell
blablabla.sh lol1 lol2
The shell recognizes that you are trying to execute something so it invokes the kernel
exec blablabla.sh lol1 lol2
The kernel finds the shebang, and it becomes
exec /bin/bash blablabla.sh lol1 lol2
By exec i mean one of the exec family system calls.
Other fun names for #! include sha-bang, hashbang, pound-bang, hash-exclam and hash-pling
Because the .sh script extension is completely arbitrary in Unix environments. Its only a convention. You can name your script whatever you like so long as it has an executable bit set. This is unlike Windows where I believe its required (.com, .exe, and I think .dll).
touch just changes the timestamp of the file. It again does not care what the file extension of the file is. In fact most tools in Unix do not care about file extension.
I thought about a concurrency issue (in Solaris), what happen if while reading someone tries to delete the same file. I have a query regarding file existence in the Solaris/Linux. suppose I have a file test.txt, I have open it in vi editor, and then I have open a duplicate session and remove that file, but even after deleting that file I am able to read that file. so here are my questions:
Do I need to thinks about any locking mechanism while reading, so no one able to delete same file while reading.
What is the reason of showing different behavior from windows(like in windows if file is open in in some editor than we can not delete that file)
After removing that file, how I am still able to read that file, if I haven't closed file from vi editor.
I am asking files in general,but yes platform specific i.e. unix. what will happen if I am using a java program (buffer reader) for read file and file is deleted while reading, does buffer reader still able to read the file for next chunk or not?
You have basically 2 or 3 unrelated questions there. Text editors like to read the whole file into memory at the start of the editing session. Imagine every character you type being saved to disk immediately, with all characters after it in the file being rewritten one place further along to make room. That would be awful. Much better that the thing you're actually editing is a memory representation of the file (array of pointers to lines, probably with some metadata attached) which only gets converted back into a linear stream when you explicitly save.
Any relatively recent version of vim will notify you if the file you are editing is deleted from its original location with the message
E211: File "filename" no longer available
This warning is not just for unix. gvim on Windows will give it to you if you delete the file being edited. It serves as a reminder that you need to save the version you're working on before you exit, if you don't want the file to be gone.
(Note: the warning doesn't appear instantly - vim only checks for the original file's existence when you bring it back into the foreground after having switched away from it.)
So that's question 1, the behavior of text editors - there's no reason for them to keep the file open for the whole session because they aren't actually using it except at startup and during a save operation.
Question 2, why do some Windows editors keep the file open and locked - I don't know, Windows people are nuts.
Question 3, the one that's actually about unix, why do open files stay accessible after they're deleted - this is the most interesting one. The answer, guaranteed to shock you when presented directly:
There is no command, function, syscall, or any other method which actually requests deletion of a file.
Underlying rm and any other command that may appear to delete a file there is the system call unlink. And it's called unlink, not remove or deletefile or anything similar, because it doesn't remove a file. It removes a link (a.k.a. directory entry) which is an association between a file and a name in a directory. (Note: ANSI C added remove as a more generic function to appease non-unix people who had no intention of implementing unix filesystem semantics, but on unix, remove is just a rmdir if the target is a directory, and unlink for everything else.)
A file can have multiple links (see the ln command for how they are created), which means that the same file is known by multiple names. If you rm one of them, the others stick around and the file is not deleted. What happens when you remove the last link? Well, now you have a file with no name. But names are only one kind of reference to a file. There are at least 2 others: file descriptors and mmap regions. When the last reference to a file goes away, that's when the file is deleted.
Since references come in several forms, there are many kinds of events that can cause a file to be deleted. Here are some examples:
unlink (rm, etc.)
close file descriptor
dup2 (can implicitly closes a file descriptor before replacing it with a copy of a different file descriptor)
exec (can cause file descriptors to be closed via close-on-exec flag)
munmap (unmap memory region)
mmap (if you create a new memory map at an address that's already mapped, the old mapping is unmapped)
process death (which closes all file descriptors and unmaps all memory mappings of the process)
normal exit
fatal signal generated by the kernel (^C, segfault)
fatal signal sent from another process (kill)
I won't call that a complete list. And I don't encourage anyone to try to build a complete list. Just know that rm is "remove name", not "remove file", and files go away as soon as they're not in use.
If you want to destroy the contents of a file immediately, truncate it. All processes already using it will find that its size has suddenly become 0. (This is destruction as far as the normal file access methods are concerned. To destroy it more thoroughly so that even someone with raw disk access can't read what used to be there, you need to overwrite it. There's a tool called shred for that.)
I think your question has nothing to do with the difference between Windows/Linux. It's about how VI works.
when using VI to edit a file. VI will create a .swp file. And the .swp file is what you are actually editing. At the same time, if other users delete the original file will not effect your editing.
And when you type :w in VI, VI will use .swp file to overwrite the original file.
I am trying to write a script or a piece of code to archive files, but I do not want to archive anything that is currently open. I need to find a way to determine what files in a directory are open. I want to use either Perl or a shell script, but can try use other languages if needed. It will be in a Linux environment and I do not have the option to use lsof. I have also had inconsistant results with fuser. Thanks for any help.
I am trying to take log files in a directory and move them to another directory. If the files are open however, I do not want to do anything with them.
You are approaching the problem incorrectly. You wish to keep files from being modified underneath you while you are reading, and cannot do that without operating system support. The best that you can hope for in a multi-user system is to keep your archive metadata consistent.
For example, if you are creating the archive directory, make sure that the number of bytes stored in the archive matches the directory. You can checksum the file contents before and after reading the filesystem and compare that with what you wrote to the archive and perhaps flag it as "inconsistent".
What are you trying to accomplish?
Added in response to comment:
Look at logrotate to steal ideas about how to handle this consistently just have it do the work for you. If you are concerned that rename of files will make processes that are currently writing them will break things, take a look at man 2 rename:
rename() renames a file, moving it
between directories if required. Any
other hard links to the file (as
created using link(2)) are unaffected.
Open file descriptors for oldpath are
also unaffected.
If newpath already exists it will be atomically replaced (subject
to a few conditions; see ERRORS
below), so that there is no point at
which another process attempting to
access newpath will find it missing.
Try ls -l /proc/*/fd/* as root.
msw has answered the question correctly but if you want to file the list of open processes, the lsof command will give it to you.
I am trying to write a bash script to do a task, I have done pretty well so far, and have it working to an extent, but I want to set it up so it's distributable to other people, and will be opening it up as open source, so I want to start doing things the "conventional" way. Unfortunately I'm not all that sure what the conventional way is.
Ideally I want a link to an in depth online resource that discusses this and surrounding topics in depth, but I'm having difficulty finding keywords that will locate this on google.
At the start of my script I set a bunch of global variables that store the names of the dirs that it will be accessing, this means that I can modify the dir's quickly, but this is programming shortcuts, not user shortcuts, I can't tell the users that they have to fiddle with this stuff. Also, I need for individual users' settings not to get wiped out on every upgrade.
Questions:
Name of settings folder: ~/.foo/ -- this is well and good, but how do I keep my working copy and my development copy separate? tweek the reference in the source of the dev version?
If my program needs to maintain and update library of data (gps tracklog data in this case) where should this directory be? the user will need to access some of this data, but it's mostly for internal use. I personally work in cygwin, and I like to keep this data on separate drive, so the path is wierd, I suspect many users could find this. for a default however I'm thinking ~/gpsdata/ -- would this be normal, or should I hard code a system that ask the user at first run where to put it, and stores this in the settings folder? whatever happens I'm going ot have to store the directory reference in a file in the settings folder.
The program needs a data "inbox" that is a folder that the user can dump files, then run the script to process these files. I was thinking ~/gpsdata/in/ ?? though there will always be an option to add a file or folder to the command line to use that as well (it processed files all locations listed, including the "inbox")
Where should the script its self go? it's already smart enough that it can create all of it's ancillary/settings files (once I figure out the "correct" directory) if run with "./foo --setup" I could shove it in /usr/bin/ or /bin or ~/.foo/bin (and add that to the path) what's normal?
I need to store login details for a web service that it will connect to (using curl -u if it matters) plan on including a setting whereby it asks for a username and password every execution, but it currently stores it plane text in a file in ~/.foo/ -- I know, this is not good. The webservice (osm.org) does support oauth, but I have no idea how to get curl to use it -- getting curl to speak to the service in the first place was a hack. Is there a simple way to do a really basic encryption on a file like this to deter idiots armed with notepad?
Sorry for the list of questions, I believe they are closely related enough for a single post. This is all stuff that stabbing at, but would like clarification/confirmation over.
Name of settings folder: ~/.foo/ -- this is well and good, but how do I keep my working copy and my development copy separate?
Have a default of ~/.foo, and an option (for example --config-directory) that you can use to override the default while developing.
If my program needs to maintain and update library of data (gps tracklog data in this case) where should this directory be?
If your script is running under a normal user account, this will have to be somewhere in the user's home directory; elsewhere, you'll have no write permissions. Perhaps ~/.foo/tracklog or something? Again, add a command line option, and also an option in the configuration file, to override this.
I'm not a fan of your ~/gpsdata default; I don't want my home directory cluttered with all sorts of directories that programs created without my consent. You see this happen on Windows a lot, and it's really annoying. (Saved games in My Documents? Get out of here!)
The program needs a data "inbox" that is a folder that the user can dump files, then run the script to process these files. I was thinking ~/gpsdata/in/ ?
As stated above, I'd prefer ~/.foo/inbox. Also with command-line option and configuration file option to change this.
But do you really need an inbox? If the user needs to run the script manually over some files, it might be better just to accept those file names on the command line. They could just be processed wherever, without having to move them to a "magic" location.
Where should the script its self go?
This is usually up to the packaging system of the particular OS you're running on. When installing from source, /usr/local/bin is a sensible default that won't interfere with package managers.
Is there a simple way to do a really basic encryption on a file like this to deter idiots armed with notepad?
Yes, there is. But it's better not to, because it creates a false sense of security. Without a master password or something, secure storage is not possible! Pidgin, for example, explicitly stores passwords in plain text, so that users won't make any false assumptions about their passwords being stored "securely". So it's best just to store them in plain text, complain if the file is world-readable, and add a clear note to the manual to warn the user what's going on.
Bottom line: don't try to reinvent the wheel. There have been thousands of scripts and programs that faced the same issues; most of them ended up adopting the same conventions, and for good reasons. Look at what they do, and mimic them instead of reinventing the wheel.
You can start with the Filesystem Hierarchy Standard. I'm not sure how well followed it is, but it does provide some guidance. In general, I try to use the following:
$HOME/.foo/ is used for user-specific settings - it is hidden
$PREFIX/etc/foo/ is for system-wide configuration
$PREFIX/foo/bin/ is for system-wide binaries
sym-links from $PREFIX/foo/bin are added to $PREFIX/bin/ for ease of use
$PREFIX/foo/var/ is where variable data would live - this is where your input spools and log files would live
$PREFIX should default to /opt/foo even though almost everyone seems to plop stuff in /usr/local by default (thanks GNU!). If someone wants to install the package in their home directory, then substitute $HOME for $PREFIX. At least that is my take on how this should all work.
When I load GVim from the Cygwin command line or when I'm not connected to the office network (mapped to U:/) it loads instantaneously. It takes a good minute or so to open when I'm on the network. What is going on here?
You probably need to set your $HOME directory to something other than U:/. Create a $HOME environment variable for Windows that points to My Documents or whatever and it won't try and use U:/. You may need to log-off/log-in to pick up the changes.
You should check what your $HOME directory is mapped to in each case. I bet that it is different depending on if you are connected to the network or not.
Do you have any custom .vimrc or other config files read at startup? If nothing else, vim will query your $HOME directory to look for such files. If your network is slow, it could cause your problem.
See ":help startup" inside vim for more details on the startup process.
It might also be a good idea to set the directory and backupdir options so they prefer a local directory. Something along the lines of
set backupdir^=C:/Temp
set directory^=C:/Temp//
The ^= syntax puts the directory at the start of the list of directories for each option. The trailing // for directory tells Vim to encode the full path to the file being edited in the swapfile's name. This allows for swapfiles to be created in one directory when editing multiple files with the same name, but different paths.