I'm trying to write a personal backup command-line utility on OSX. Let's say I have two folders:
foo/bar/
foo/baz/
foo/bar contains, among other things, OSX aliases to files in foo/baz:
foo/bar/file_alias# -> foo/baz/file
I want to copy both foo/bar and foo/baz to an external hard drive, but for various reasons I do not just want to copy the entire folder foo. I can't figure out a way to copy these folders separately and make the aliases come out right in the end:
cp -r foo/bar /external_hd/foo/bar follows the aliases, replacing them with the original files.
cp -R foo/bar /external_hd/foo/bar preserves the aliases, but they (not surprisingly) continue to point to the original files (e.g. foo/baz/file, not external_hd/foo/baz/file).
rsync -avE foo/bar /external_hd/foo/bar (see this question) seems to do the same thing as cp -R.
Is there any way to accomplish this without copying the entire parent folder foo?
I know of no way where you can automatically copy folders and relink symbolic links to a new destination without some manual intervention. If you know the new paths its quite simple to script, though.
For your specific example; the following should do the trick to relink:
cd /external_hd/foo
find . -type l | while read x; do y=$(readlink "$x" | sed s'|/foo|/external_hd/foo|'); ln -sf "$y" "$x";done
rsync will get you close, the command:
rsync -avHER --safe-links foo/{bar,baz} /external_hd/
will copy the two folders, preserve "safe" relative symlinks between, and ignore "unsafe" symlinks - those that may reference files outside of the copied tree. Change it to:
rsync -avHER --copy-unsafe-links foo/{bar,baz} /external_hd/
and "safe" relative symlinks are preserve and "unsafe" symlinks are replaced by their destination.
If you only have "safe" relative symlinks the first option will do, the second option may do if some extra copying is OK.
However, the definition of "safe" is over-restrictive. Any absolute symlink is "unsafe" even if its target is within the copied tree. Furthermore even a relative link which goes too far towards the root, or maybe is just too complicated, is also "unsafe".
If you need to fix this it should be possible, as the above options show rsync is pretty close to what you need and the source code is available from Apple's Open Source site. Examine the code around the options --links, --copy-links, --copy-unsafe-links & unsafe-links and you may find fixing the definition of "safe" is fairly easy (and you can re-write the symlinks to use the shortest possible relative path at the same time).
HTH
Related
I have two directory structures on a USB drive that have various files that are the same, and each have files that the other one doesn't.
What I want to do is to move over directory structure B to A. It is ok that B's content is gone after the merge. Directories in A must not be erased, because otherwise I lose A's content. The mv command won't work I think, because it will complain that it can't move a directory because a destination directory in the same place is not empty. mv B/* A/ won't work either because some sub directory will also not be empty.
cp -a B/* A/ is bad (even with -u), because it will take way too long, because the files are on a USB drive, and there possibly too many of them, making the drive run out of capacity.
rsync has the same problem, because it doesn't appear to have a move/rename feature, and it can only move files by copying them.
So either, I'm going to have to write a script that will recursively run through B, and create missing directories and move missing files to A.
But I'm hoping that there is a command or option or utility that I don't know about.
I believe cpio has the capabilities you are wanting. This command:
cd B
find . -type f -print0 | cpio -0dumpl A/.
Will find all files in B, pass them to cpio with null termination to properly handle odd file names, create necessary directories (cpio -d), preserve ownership, permissions and timestamps (-m), and use linking to create the destination files where possible (-l) unconditionally (-u).
This is a basic question but I am struggling to find a decent solution. This is hindering my script from automation.
I have the following path.
/home/hassan/Dyna/ProjectSimulation
in project simulation I have 3 folders
friction time force
like
/home/hassan/Dyna/ProjectSimulation/friction
Now I have a file friction1.txt in this friction folder and I want to copy it to ProjectSimulation.
is it possible to avoid complete path and just one step down?
Also if I have to copy this friction1.txt to folder force, is there anyway to avoid the complete path.
I mean I have a subroutine but this is path dependent , whenever I run it , I have to run in the same folder and then copy my results so I can run only one instance of my simulation.
Experts please guide me.
PS: This is part of a 600 lines shell.
This comes across as so basic that I must have misunderstood something in your question.
If you want to refer to a parent directory, .. is the way to do that. So, if you want to copy friction1.txt to two places you just do
cp friction1.txt ..
cp friction1.txt ../force
All you need to take care of is making sure that CWD is
/home/hassan/Dyna/ProjectSimulation/friction
so that the references point at the right place.
You can temprarily change the current directory to ProjectSimulation, copy the file (cp friction/friction1.txt .), then change the path back to the original (so the rest of the script works as before)
Alternatively, you can use dirname to get he name of the parent directory and use that.
Change to the root dir of your known directory structure. Then do the copy operations with relative paths. Then change back to your dir where you came from.
Your friends are:
cd
cd -
or better:
pushd
popd
(see man bash)
I.e.
pushd /home/hassan/Dyna/ProjectSimulation
cp friction/friction1.txt .
cp friction/friction1.txt force
popd
the basic idea is that I want to link to path that's relative to $HOME, rather than explicitly expand the $HOME variable, as I want to make sure the link works on multiple machines, e.g.,
when I do
ln -s ~/data datalnk
I want it to be directed to directory /home/user/data on one machine which has a user $HOME of /home/user, and to /home/machine/user/data on another machine which has a user $HOME of /home/machine/user/data.
I cannot create a symbolic link on the second machine using
ln -s /home/machine/user /home/user
because I don't have the permission to do that, and I cannot link relative paths as the two machines have different hierarchies of directories.
anyideas on possible ways to fix or circumvent this?
EDIT:
what I am really trying to accompanish is to make the same link work on two macihnes, where the targets have the same directories in terms of their relative path to $/HOME only, not their absolute path, and not their relative path to the link either.
tl,dr it won't work
You can use an escaping mechanism such as single-quotes to get the ~ into the symbolic link:
> cd ~
> echo hello > a
> ln -s '~/a' b
However, ~ is a shell expansion and is not understood by the filesystem (actually, to the filesystem it's "just another character"). This is a good thing -- want the file-system layer to know about environment variables, as ~ is generally determined by $HOME?
> ls -l b
lrwxrwxrwx 1 root root 3 Oct 27 17:39 b -> ~/a
> ls b
ls: b: No such file or directory
You could still "manually" look at said symbolic link entries (as done with ls -l), but that would have to be done in a non-transparent fashion by a program (think of a ".LNK" in Windows). As can be seen, the filesystem just doesn't understand ~.
Happy sh'ing.
First of all: It can't be done directly. Symbolic links are plain text files, no extensions are performed. If you can't formulate a fixed relative or absolute path to the place you are referring, you can't symbolically link to it.
You can build a script to put links to appropriate directories in appropriate places, but the best way depends on your application.
The only way to make symlinks dynamic in this way is to use a relative path instead of an absolute path. In other words, don't start your path with /.
For example:
cd
ln -s data datalnk
At runtime your app or script will need to refer to ~/datalnk or $HOME/datalnk.
You haven't really said what you're trying to accomplish, so I can't really tell whether I'm solving your problem or suggesting that you need to go at it a different way.
I'm trying to use something along the lines of
unexpand -t 4 *.php
but am unsure how to write this command to do what I want.
Weirdly,
unexpand -t 4 file.php > file.php
gives me an empty file. (i.e. overwriting file.php with nothing)
I can specify multiple files okay, but don't know how to then overwrite each file.
I could use my IDE, but there are ~67000 instances of to be replaced over 200 files, and this will take a while.
I expect that the answers to my question(s) will be standard unix fare, but I'm still learning...
You can very seldom use output redirection to replace the input. Replacing works with commands that support it internally (since they then do the basic steps themselves). From the shell level, it's far better to work in two steps, like so:
Do the operation on foo, creating foo.tmp
Move (rename) foo.tmp to foo, overwriting the original
This will be fast. It will require a bit more disk space, but if you do both steps before continuing to the next file, you will only need as much extra space as the largest single file, this should not be a problem.
Sketch script:
for a in *.php
do
unexpand -t 4 $a >$a-notab
mv $a-notab $a
done
You could do better (error-checking, and so on), but that is the basic outline.
Here's the command I used:
for p in $(find . -iname "*.js")
do
unexpand -t 4 $(dirname $p)/"$(basename $p)" > $(dirname $p)/"$(basename $p)-tab"
mv $(dirname $p)/"$(basename $p)-tab" $(dirname $p)/"$(basename $p)"
done
This version changes all files within the directory hierarchy rooted at the current working directory.
In my case, I only wanted to make this change to .js files; you can omit the iname clause from find if you wish, or use different args to cast your net differently.
My version wraps filenames in quotes, but it doesn't use quotes around 'interesting' directory names that appear in the paths of matching files.
To get it all on one line, add a semi after lines 1, 3, & 4.
This is potentially dangerous, so make a backup or use git before running the command. If you're using git, you can verify that only whitespace was changed with git diff -w.
I am looking to do a specific copy in Fedora.
I have two folders:
'webroot': holding ALL web files/images etc
'export': folder containing thousands of PHP, CSS, JS documents that are exported from my SVN repo.
The export directory contains many of the same files/folders that the root does, however the root contains additional ones not found in export.
I'd like to merge all of the contents of export with my webroot with the following options:
Overwriting the file in webroot if export's version contains different code than what
is inside of webroot's version (live)
Preserve the permissions/users/groups of the file if it is overwritten (the export
version replacing the live version) *NOTE I would like the webroots permissions/ownership maintained, but with export's contents
No prompting/stopping of the copy
of any kind (ie not verbose)
Recursive copy - obviously I
would like to copy all* files
folders and subfolders found in
export
I've done a bit of research into cp - would this do the job?:
cp -pruf ./export /path/to/webroot
It might, but any time the corresponding files in export and webroot have the same content but different modification times, you'd wind up performing an unnecessary copy operation. You'd probably get slightly smarter behavior from rsync:
rsync -pr ./export /path/to/webroot
Besides, rsync can copy files from one host to another over an SSH connection, if you ever have a need to do that. Plus, it has a zillion options you can specify to tweak its behavior - look in the man page for details.
EDIT: with respect to your clarification about what you mean by preserving permissions: you'd probably want to leave off the -p option.
-u overwrites existing files folder if the destination is older than source
-p perserves the permission and dates
-f turns off verbosity
-r makes the copy recursive
So looks like you got all the correct args to cp
Sounds like a job for cpio (and hence, probably, GNU tar can do it too):
cd export
find . -print | cpio -pvdm /path/to/webroot
If you need owners preserved, you have to do it as root, of course. The -p option is 'pass mode', meaning copy between locations; -v is verbose (but not interactive; there's a difference); -d means create directories as necessary; -m means preserve modification time. By default, without the -u option, cpio won't overwrite files in the target area that are newer than the one from the source area.