If I rsync a directory with some remote directory under an sshfs
management, even if the rsync process finishes synching the
directory contents, there seems to be a time lag for the sshfs
directory to get updated. It is unintuitive and makes some
automation process relying on sshfs contents unstable.
How do I eliminate the lag?
-o cache=no in your sshfs options will turn off caching.
Note: no should be lower case: -o cache=NO should give you an error!
Related
I wrote a little program, where i mounted an encrypted volume after the user inserts the password with veracrypt an showed the content to the user in a specific way inside my programm. Everything works fine, but i want to prevent that the volume is shown in Nautilus.
The following command mounts the volume:
"veracrypt -m ro /path/to/file/file -p" + pw
Veracrypt help command shows:
--fs-options=OPTIONS
Filesystem mount options. The OPTIONS argument is passed to mount(8)
command with option -o when a filesystem on a VeraCrypt volume is mounted.
This option is not available on some platforms.
But i'm not able to find a mount option for linux-mount command, which will do the job. Is there any? What can i do?
A simple workaround may be to mount the volume to a directory that begins with a .. These directories should not be visible in most file managers without specifying they should show hidden files/folders.
Of course, obscuring the folder isn't perfect and is no substitute for setting proper access control specifies via chmod.
I am hoping that a more experienced set of eyes will find something obvious that I am missing or will be able to help me work around the errors that mv and rsync are producing. Up for the challenge?
Basic idea:
I have a bash script in which I am automating the move of files from one directory to another.
The problem:
When I run the script, periodically I get the following error from the mv command:
mv: cannot stat `/shares/directory with spaces/test file.txt': No such file or directory. The error code from the vm command is 1. Even more odd, the file move actually succeeds sometimes.
In addition, I have a branch of logic in the script that will alternately use rsync to move/copy specific files (from the same local file system source and destination as the mv command mentioned above). I get a similar error related to the stat() system call:
rsync: link_stat "/shares/directory with spaces/test file.txt" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]
This error does not always manifest itself when the script is run. Sometimes it completes the file move without complaint, while other times it will return the error consistently when the script is run successive times.
There is one additional ingredient you should be aware of (and I am growing to suspect this as a key ingredient in my grief): the directory /shares/ is a directory that is being monitored by an installation of Dropbox -- meaning it is watched and mirrored by an installation of Dropbox. At this point, I am unable to determine if dropboxd is somehow locking the file, or the like, such that it cannot be stat-ed. To be clear, the files are eventually freed from this state without further intervention and are mv-able.
The code:
mv -v --no-clobber "${SOURCEPATH}${item}" "${DESTINATIONPATH}${item}"
More info:
The following might, or might not, be relevant:
mount indicates the filesystem is ext4
Presumably, ownership and permissions shouldn't be an issue as the script is being run by root. Especially if the file system is not fuse-based.
The base "directory" in the path (e.g. /shares/) is a symlink to another directory on the same file system.
The flavor of Linux is Debian.
Troubleshooting:
In trying to eliminate any issues with the variable expansion or their contents, I tried hardwiring the bash script like such:
mv -v --no-clobber "/shares/directory with spaces/test file.txt" "/new destination/directory with spaces/test file.txt" after verifying via ls -al that "test file.txt" existed. For reference the permissions were: -rw-r--r--
Unfortunately, this too results in the same error.
Other possible issues I could think of and what I have done to try to rule them out:
>> possible issue: slow HDD (or drive is in low power mode) or external USB drive
>> findings: The drives are all local SATA disks set to not park heads. In addition, even when forcing a consistent read from the file system, the same error happens
>> possible issue: non-Linux, NFS or fuse-based file system
>> findings: nope, source and destination are on the same local file system and mount indicates the file system is ext4
>> possible issue: white space or other unprintable chars in the file path
>> findings: verified that the source and destination paths where properly wrapped in quotes
>> possible issue: continuation issues after escaped newline (space after \ in wrapped command)
>> findings: made sure the command was all on one line, still the same error
>> possible issue: globbing (use of * in specifying the files to move)
>> findings: nope, each file is specified directly by path and name
>> possible issue: path confusion from the use of local path
>> findings: nope, file paths are fully qualified starting from /
>> possible issue: the files are not actually in the path specified
>> findings: nope, verified the file existed right prior to executing the script via ls -al
>> possible issue: somehow the --no-clobber of mv was causing issues
>> findings: nope, tried it without, same error
>> possible issue: only files created via Dropbox sync to the file system are problematic
>> findings: nope, created a local file directly via touch new-local-file.txt and it too produced the same stat() error
My analysis:
The fact that mv and rsync produce similar stat() errors leads me to believe:
there is some systemic underlying boundary case (e.g. file permissions/ownership or file busy) that is not accounted for in the bash script; or
the same bug is plaguing me in both the mv and the rsync scenarios.
Desired outcomes:
1. The root cause of the intermittent errors can be identified.
2. The root cause can be resolved or worked around.
3. The bash script can be improved to gracefully handle when the error occurs.
So, with a lot more troubleshooting I found an errant rsync statement some 200 lines earlier in the script that was conditionally executed (thus the seeming inconsistent behavior). That rsync --archive ... statement was being passed /shares/ as its source directory, therefore it effected the /shares/directory with spaces/ subdirectory. That subdirectory was the ${SOURCEPATH} of the troubling mv command mentioned in my post above.
Ultimately, it was a missing --dry-run flag on the rsync --archive ... statement that causing the trampling of the files that the script later expected to pass to mv to process.
Thanks for all who took the time to read my post. Though I am bummed to have spent my and your time on what turned out to be a bug in my script, it is reassuring to know that:
- computers are not irrational
- I am not insane
- there is not some nefarious, deep rooted bug in the linux file system
For those that stumble upon this post in the future because you are experiencing an error of cannot stat, please read my troubleshooting notes above. Much research went into that list. One of those might be your issue. If not, keep debugging, there is an explanation. Good luck!
I am trying to do a soft link from one directory to another, the directory I am trying to access I have read and execute. However, its parent directory I do NOT have execute permissions.
Is there a way to do a soft link, to my desired directory without giving me execute permission to the parent directory?
Below is the code I used:
ln -s /home/dir1/dir2/desired_directory symbolic_link_name
the link just comes up as red with grey background.
Thank you.
Although this is not possible with symlinks, you could do it with mount --bind. Note that if the whole point is to circumvent security, then this is probably a very bad idea.
Your command would be
mount --bind /home/dir1/dir2/desired_directory mount_dir
There are a few issues to be aware of:
The target directory mount_dir must exist before (same as any mount point)
Root access is required to execute the mount commmand
The created "link" will not persist after a reboot unless a corresponding line is added to /etc/fstab
If the origin directory contains mounted file systems, these will not be transferred to the target. The mount points will appear as empty directories.
Using mount --bind may be considered bad practice because most programs are not aware that the "link" is not a standard directory. For instance it allows the creation of loops in the directory tree which make any tree parsing application (think "ls -R") enter a possibly infinite loop.
It may be hazardous when combined with recursive delete operations. See for instance Yet another warning about mount --bind and rm -rf.
Symbolic links are not a way to circumvent permissions set on their targets. No, there is no way to do what you want. If it was possible it would be a serious security issue.
I am using rsync to backup a million images from my linux server to my computer (windows 7 using Cygwin).
The command I am using now is :
rsync -rt --quiet --rsh='ssh -p2200' root#X.X.X.X:/home/XXX/public_html/XXX /cygdrive/images
Whenever the process is interrupted, and I start it again, it takes long time to start the copying process.
I think it is checking each file if there is any update.
The images on my server won't change once they are created.
So, is there any faster way to run the command so that it may copy files if local file doesn't exist without checking filesize, time, checksum etc...
Please suggest.
Thank you
did you try this flag -- it might help, but it might still take some time to resume the transfer:
--ignore-existing
This tells rsync to skip updating files that already exist on the destination (this does not ignore
existing directories, or nothing would get done). See also --existing.
This option is a transfer rule, not an exclude, so it doesn't affect the data that goes into the
file-lists, and thus it doesn't affect deletions. It just limits the files that the receiver requests
to be transferred.
This option can be useful for those doing backups using the --link-dest option when they need to con-
tinue a backup run that got interrupted. Since a --link-dest run is copied into a new directory hier-
archy (when it is used properly), using --ignore existing will ensure that the already-handled files
don't get tweaked (which avoids a change in permissions on the hard-linked files). This does mean that
this option is only looking at the existing files in the destination hierarchy itself.
I have a statically linked code(not a module) in kernel that should launch kernel thread after root file system is mounted. The problem is I don't know how to do this without modifying prepare_namespace() kernel function. I thought it's possible to do via initcalls but
they're executed before kernel takes care about rootfs.
Does anyone know the best way to do this?
UPDATE [1]: #BenVoigit suggested the following solution in comments:
Seems like you should open /proc/mounts and poll_wait on it. See the source for `mounts_poll'
UPDATE [2]: I looked at RSBAC patches, RSBAC modifies prepare_namespace() function to make some actions after filesystem is mounted. It seems to be the easiest way.
Well, current Linux images are too big to fit the PC boot sector. Modern bootloaders like grub will mount an small filesystem in RAM before the real one.
To understand what is happening under the hood, you can open the disk image located under /boot. For example, in Ubuntu:
mkdir test
cd test
zcat /boot/initrd.img-2.6.35-24-generic > image.cpio
cpio -i < image.cpio
vim init
In the end, it's just a bunch of shell scripts - the simplicity is almost poetic.